Tags

scripts: mandate use of SPDX-License-Identifier tags in new files

One of the items raised at the QEMU maintainers meeting at KVM Forum
2024 was adoption of SPDX-License-Identifier for licensing of newly
contributed source files, for which there were no dissenting voices.

Thus, this series proposes a way to put this into action by extending
checkpatch.pl to mandate SPDX-License-Identifier in all new files.

Furthermore, anytime it sees SPDX-License-Identifier in any patch,
whether a new file or pre-existing, it validates the declared license
name. If it is not one of the commonly used QEMU licenses (the GPL
variants, MIT, & a few BSD variants), it will report an error. To
encourage sticking with GPL-2.0-or-later by default, it will issue
a warning even if it is one of the common licenses, encouraging
the contributor to double check their choice. This will reduce
accidental license proliferation.

Finally, I've seen a few other random SPDX tags such as:

  * SPDX-FileCopyrightText  - replacing "Copyright ..."
  * SPDX-FileContributor - replacing "Authors: ..."
  * SPDX-URL - a link to the link license text
  * SPDX-sourceInfo - arbitrary free form text about the file

These may or may not be worth considering in QEMU, but this series
discourages their usage by raising an error in checkpatch for now.

If we feel we want to adopt any of these, I think it should be
through a concious decision applied universally. Inconsistent &
adhoc usage of other SPDX tags by a subset of contributors feels
like it doesn't seem to give a clear win, and could even be a
net loss through making practices inconsistent across the code.

Changed in v3:

 * Add missing accepted license LGPL-2.1-only
 * Drop LGPL-2.0-only & LGPL-2.0-or-later as acceptable
 * Fix typo in commit message

Changed in v2:

 * Tweaks to the commit messages
 * Expand the message warning about non GPL-2.0-or-later
   usage, to request an explanation in the commit message
   for the unusual choice.

* Remove deprecated 'loaded' property from crypto objects
* Fix error checking of hash function in gcrypt
* Perform runtime check for hash functions in gcrypt
* Add SM3 hash function to pbkdf

Misc sockets, crypto and VNC fixes

* Fix rare EADDRINUSE failures on OpenBSD platforms seen
  with migration
* Fix & test overwriting of hash output buffer
* Close connection instead of returning empty SASL mechlist to
  VNC clients
* Fix handling of SASL SSF on VNC server UNIX sockets
* Fix handling of NULL SASL server data in VNC server
* Validate trailing NUL padding byte from SASL client
* Fix & test AF_ALG crypto backend build
* Remove unused code in sockets and crypto subsystems

Convert avocado tests to normal Python unittests

This series is an update to Thomas' v2:

  https://lists.nongnu.org/archive/html/qemu-devel/2024-07/msg05805.html

wherein Thomas suggested someone could do the asset caching updates
while he is on vacation:

  https://lists.nongnu.org/archive/html/qemu-devel/2024-07/msg06228.html

hence this posting what I'm calling a v3.

The Avocado v88 that we use in QEMU is already on a life support
system: It is not supported by upstream anymore, and with the latest
versions of Python, it won't work anymore since it depends on the
"imp" module that has been removed in Python 3.12.

There have been several attempts to update the test suite in QEMU
to a newer version of Avocado, but so far no attempt has successfully
been merged yet.

Additionally, the whole "make check" test suite in QEMU is using the
meson test runner nowadays, so running the python-based tests via the
Avocodo test runner looks and feels quite like an oddball, requiring
the users to deal with the knowledge of multiple test runners in
parallel (e.g. the timeout settings work completely differently).

So instead of trying to update the python-based test suite in QEMU
to a newer version of Avocado, we should try to better integrate
it with the meson test runner instead. Indeed most tests work quite
nicely without the Avocado framework already, as you can see with
this patch series - it does not convert all tests, just a subset so
far, but this already proves that many tests only need small modifi-
cations to work without Avocado.

Only tests that use the LinuxTest / LinuxDistro and LinuxSSHMixIn
classes (e.g. based on cloud-init images or using SSH) really depend
on the Avocado framework, so we'd need a solution for those if we
want to continue using them. One solution might be to simply use the
required functions from avocado.utils for these tests, and still run
them via the meson test runner instead, but that needs some further
investigation that will be done later.

Now if you want to try out these patches: Apply the patches, then
recompile and then run:

 make check-functional

You can also run single targets e.g. with:

 make check-functional-ppc

You can also run the tests without any test runner now by
setting the PYTHONPATH environment variable to the "python" folder
of your source tree, and by specifying the build directory via
QEMU_BUILD_ROOT (if autodetection fails) and by specifying the
QEMU binary via QEMU_TEST_QEMU_BINARY. For example:

 export PYTHONPATH=$HOME/qemu/python
 export QEMU_TEST_QEMU_BINARY=qemu-system-x86_64
 export QEMU_BUILD_ROOT=$HOME/qemu/build
 ~/qemu/tests/functional/test_virtio_version.py

The logs of the tests can be found in the build directory under
tests/functional/<arch>/<testname> - console log and general logs will
be put in separate files there.

Still to be done: Update the documentation for this new test framework.

v3:
- Split tests/functional/__init__.py into multiple files
- Introduce an 'Asset' class to handle downloading & caching
- Convert all tests to use new Asset class instead of 'fetch_asset'
- Ensure 'make check-functional' pre-caches all assets before
  running the tests, to avoid meson test timeouts due to slow
  downloads
- Fix bios bits test logging in verbose mode to not print subprocess
  output to stdout, since it confuses TAP parsing

v2:
- Addressed review feedback from v1
- Add pycotap as a wheel instead of trying to install it on demand
  when running "make check-functional" (works much better now!)
- Converted much more tests
- Lots of other small improvements here and there

RFC -> v1:
- Now using pycotap for running the tests instead of "pytest"
- Change the name from "tests/pytest" to "tests/functional" accordingly
- Make it possible to run the tests directly
- Use Python's urllib instead of wget for downloading
- Lots of makefile / meson integration improvements
- Converted more tests
- Update MAINTAINERS file accordingly
- Added a patch to run check-functional in the gitlab-CI
- ... lots of other changes I forgot about ... in fact, I changed so
  many things that I also did not dare to pick up the Reviewed-bys
  from the RFC

qga: clean up command source locations and conditionals

This series is a side effect of other work I started, to attempt to
make the QGA safe to use in confidential VMs by automatically
restricting the permitted commands. Since this cleanup stands on
its own, I'm sending it now.

The QGA codebase has a very complicated maze of #ifdefs to create
stubs for the various commands that cannot be implemented on certain
platforms. It then has further logic to dynamically disable the stub
commands at runtime, except this is not consistently applied, so
some commands remain enabled despite being merely stubs.

The resulting code is hard to follow, when trying to understand exactly
what commands are available under what circumstances, and when changing
impls it is easy to get the #ifdefs wrong, resulting in stubs getting
missed on platforms without a real impl. In some cases, we have multiple
stubs for the same command, due to the maze of #ifdefs.

The QAPI schema language has support for many years for expressing
conditions against commands when declaring them. This results in the
QAPI code generator omitting their implementation entirely at build
time. This has mutliple benefits

 * The unsupported commands are guaranteed to not exist at runtime
 * No stubs need ever be defined in the code
 * The generated QAPI reference manual documents the build conditions

This series is broadly split into three parts

 * Moving tonnes of Linux only commands out of commands-posix.c
   into commands-linux.c to remove many #ifdefs.
 * Adding 'if' conditions in the QAPI schema to reflect the
   build conditions, removing many more #ifdefs
 * Sanitizing the logic for disabling/enabling commands at
   runtime to guarantee consistency

Changed in v2:

 - Make FSFreeze error reporting distinguish inability to enable
   VSS from user config choice

 - Fully remove ga_command_init_blockedrpcs() methods. No more
   special case disabling of commands. Either they're disabled
   at build time, or disabled by user config, or by well defined
   rule ie not permitted during FS freeze.

 - Apply rules later in startup to avoid crash from NULL config
   pointer

 - Document changed error messages in commit messages

 - Add -c / --config command line parameter

 - Fix mistaken enabling of fsfreeze hooks on win32

Improve mechanism for configuring allowed commands

The QGA supports dynamically filtering what commands are enabled via a
combination of allow lists and deny lists. This is very flexible, but
at the same time very fragile.

Consider that a user wants to block all commands that allow unrestricted
file access/command execution, so they set the deny list when starting
QGA. Now their OS vendor issues a software update which includes a new
version of QGA. This new QGA version is liable to contain new commands,
some of which might undermine the intent of the user's configured deny
list.

IOW, the generic deny list functionality is inherently dangerous as a
mechanism for limiting risk exposure.

Using an allow list is much safer, but means on every update the user
should check the list of new commands to decide which are safe or not,
putting a burden on every user.

In the context of RHEL, there has been a long term deny list that blocks
use of guest-file and guest-exec commands, since they give unrestricted
access to the guest.

With the advent of confidential computing, a far greater number of QGA
commands are very unsafe to permit, and it is unreasonable to expect
each user and/or downstream vendor to repeat the work to figure out
what commands are OK.

This is a similar problem seen in the "seccomp" world where new syscalls
appear frequently and users can't be expected to understand all of them.
Systemd pioneered the approach of defining "profiles"  which group
together sets of syscalls, which we subsequently copied in QEMU.

This series applies this same conceptual idea to QGA command filtering,
making use of the QAPI "features" facility to associate commands into
one or more groups.

This grouping is then exposed via some new higher level command line
arguments.

* --no-unrestricted / -u

  A flag to block all the guest-file and guest-exec commands

  This replicates the policy RHEL currently defines via a deny list.

* --no-user-auth / -e

  A flag to block all the commands for manipulating user account
  authentication credentials.

* --confidential / -i

  A flag to block all commands, except for those which have been
  explicitly marked as not violating guest owner data privacy

This feature mechanism is further utilized internally to track the
commands which are safe to use while FS are frozen.

A key benefit of using the QAPI "features" facility is that these
groupings are visible in the documentation of the QGA commands.

By using these high level command lines arguments, deployments will
be safe wrt software upgrades, as long as QEMU maintainers apply
appropriate tags to any new commands.

The allow/deny list command line flags can still be used to further
refine the command lines, but ideally that would be rare.

A missing piece in this series is getting the --confidential flag to
be automatically passed to QGA when running in a confidential VM. This
is something that will likely be done via systemd unit files. My thought
is that the existing 'qemu-guest-agent.service' would get a parameter

   ConditionSecurity=!cvm

while a new qemu-guest-agent-confidential.service' would have the same
content but with ConditionSecurity=cvm instead, and would pass the
--confidential flag.

This series depends on the one I sent earlier:

  https://lists.nongnu.org/archive/html/qemu-devel/2024-06/msg00743.html

Fix crash during early close of TLS channel

Miscellaneous fixes

 * Switch canokey license from Apache to GPLv2+
 * Fix uninitialized variable in LUKS driver

Merge asymmetric cipher crypto support

This extends the internal crypto APIs to support the use of asymmetric
ciphers.

Add property for requesting AMD SEV measured kernel launch

 - The 'sev-guest' object gains a boolean 'kernel-hashes' property
   which must be enabled to request a measured kernel launch.

Initial conversion of HMP debugging commands to QMP

This introduces a new policy that all HMP commands will be converted to
have QMP equivalents, marked unstable if no formal QAPI modelling is
intended to be done.

New unstable commands are added as follows:

  - HMP "info roms" => QMP "x-query-roms"
  - HMP "info profile" => QMP "x-query-profile"
  - HMP "info numa" => QMP "x-query-numa"
  - HMP "info usb" => QMP "x-query-usb"
  - HMP "info rdma" => QMP "x-query-rdma"
  - HMP "info ramblock" => QMP "x-query-ramblock"
  - HMP "info irq" => QMP "x-query-irq"
  - HMP "info jit" => QMP "x-query-jit"
  - HMP "info opcount" => QMP "x-query-opcount"

Merge crypto updates and misc fixes

 * Introduce a GNUTLS backend for crypto algorithms
 * Change crypto library preference gnutls > gcrypt > nettle > built-in
 * Remove built-in DES impl
 * Remove XTS mode from built-in AES impl
 * Fix seccomp rules to allow resource info getters
 * Fix migration performance test
 * Use GDateTime in io/ and net/rocker/ code
 * Improve docs for -smp

Hide build time dependancy on gnutls fom non-crypto code

Remove many old deprecated features

The following features have been deprecated for well over the 2
release cycle we promise

  ``-drive file=json:{...{'driver':'file'}}`` (since 3.0)
  ``-vnc acl`` (since 4.0.0)
  ``-mon ...,control=readline,pretty=on|off`` (since 4.1)
  ``migrate_set_downtime`` and ``migrate_set_speed`` (since 2.8.0)
  ``query-named-block-nodes`` result ``encryption_key_missing`` (since 2.10.0)
  ``query-block`` result ``inserted.encryption_key_missing`` (since 2.10.0)
  ``migrate-set-cache-size`` and ``query-migrate-cache-size`` (since 2.11.0)
  ``query-named-block-nodes`` and ``query-block`` result dirty-bitmaps[i].status (since 4.0)
  ``query-cpus`` (since 2.12.0)
  ``query-cpus-fast`` ``arch`` output member (since 3.0.0)
  ``query-events`` (since 4.0)
  chardev client socket with ``wait`` option (since 4.0)
  ``acl_show``, ``acl_reset``, ``acl_policy``, ``acl_add``, ``acl_remove`` (since 4.0.0)
  ``ide-drive`` (since 4.2)
  ``scsi-disk`` (since 4.2)

 - Fix inverted logic in abstract socket QAPI support
 - Only report abstract socket support in QAPI on Linux hosts
 - Expand test coverage
 - Misc other code cleanups

Improve performance of crypto cipher subsystem

v5.0.0 release

v5.0.0-rc4 release

v5.0.0-rc3 release