This commit removes the PYTHON_UTF8 workaround. The problem with setting
LC_ALL= LANG=C LC_CTYPE=en_US.UTF-8
is that the en_US.UTF-8 locale might not be available. In this case
setting above locales results in build errors even though another UTF-8
locale was originally set [1]. The only stable way of fixing the
encoding problem is by specifying the encoding in Python, like the
previous commit does.
[1] https://bugs.gentoo.org/657766
Backports commit 0d6b93deeeb3cc190692d629f5927befdc8b1fb8 from qemu
To be more accurate on its purpose and make code that looks for a certain
target out of this variable more readable.
Backports commit 208ecb3e1acc8d55dab49fdf721a86d513691688 from qemu
We used to generate first test and later QGA QAPI code into
qapi-generated/. Commit b93b63f574 moved the test code to tests/.
Commit 54c2e50205 moved the QGA code to qga/qapi-generated/. The
directory has been unused since.
Backports commit 418b1d0ae3a2cc992659f626a2a3f11944e0b259 from qemu
Move qapi-schema.json to qapi/, so it's next to its modules, and all
files get generated to qapi/, not just the ones generated for modules.
Consistently name the generated files qapi-MODULE.EXT:
qmp-commands.[ch] become qapi-commands.[ch], qapi-event.[ch] become
qapi-events.[ch], and qmp-introspect.[ch] become qapi-introspect.[ch].
This gets rid of the temporary hacks in scripts/qapi/commands.py,
scripts/qapi/events.py, and scripts/qapi/common.py.
Backports commit eb815e248f50cde9ab86eddd57eca5019b71ca78 from qemu
The previous commit improved compile time by including less of the
generated QAPI headers. This is impossible for stuff defined directly
in qapi-schema.json, because that ends up in headers that that pull in
everything.
Move everything but include directives from qapi-schema.json to new
sub-module qapi/misc.json, then include just the "misc" shard where
possible.
It's possible everywhere, except:
* monitor.c needs qmp-command.h to get qmp_init_marshal()
* monitor.c, ui/vnc.c and the generated qapi-event-FOO.c need
qapi-event.h to get enum QAPIEvent
Perhaps we'll get rid of those some other day.
Adding a type to qapi/migration.json now recompiles some 120 instead
of 2300 out of 5100 objects.
Backports commit 112ed241f5d9a411dbca92bdf597151cb853c6a7 from qemu
Our qapi-schema.json is composed of modules connected by include
directives, but the generated code is monolithic all the same: one
qapi-types.h with all the types, one qapi-visit.h with all the
visitors, and so forth. These monolithic headers get included all
over the place. In my "build everything" tree, adding a QAPI type
recompiles about 4800 out of 5100 objects.
We wouldn't write such monolithic headers by hand. It stands to
reason that we shouldn't generate them, either.
Split up generated qapi-types.h to mirror the schema's modular
structure: one header per module. Name the main module's header
qapi-types.h, and sub-module D/B.json's header D/qapi-types-B.h.
Mirror the schema's includes in the headers, so that qapi-types.h gets
you everything exactly as before. If you need less, you can include
one or more of the sub-module headers. To be exploited shortly.
Split up qapi-types.c, qapi-visit.h, qapi-visit.c, qmp-commands.h,
qmp-commands.c, qapi-event.h, qapi-event.c the same way.
qmp-introspect.h, qmp-introspect.c and qapi.texi remain monolithic.
The split of qmp-commands.c duplicates static helper function
qmp_marshal_output_str() in qapi-commands-char.c and
qapi-commands-misc.c. This happens when commands returning the same
type occur in multiple modules. Not worth avoiding.
Since I'm going to rename qapi-event.[ch] to qapi-events.[ch], and
qmp-commands.[ch] to qapi-commands.[ch], name the shards that way
already, to reduce churn. This requires temporary hacks in
commands.py and events.py. Similarly, c_name() must temporarily
be taught to munge '/' in common.py. They'll go away with the rename.
Backports commit 252dc3105fc494182e236e97fe20f2d6b1d652cb from qemu
Linking code from multiple separate QAPI schemata into the same
program is possible, but involves some weirdness around built-in
types:
* We generate code for built-in types into .c only with option
--builtins. The user is responsible for generating code for exactly
one QAPI schema per program with --builtins.
* We generate code for built-in types into .h regardless of
--builtins, but guarded by #ifndef QAPI_VISIT_BUILTIN. Because all
copies of this code are exactly the same, including any combination
of these headers works.
Replace this contraption by something more conventional: generate code
for built-in types into their very own files: qapi-builtin-types.c,
qapi-builtin-visit.c, qapi-builtin-types.h, qapi-builtin-visit.h, but
only with --builtins. Obey --output-dir, but ignore --prefix for
them.
Make qapi-types.h include qapi-builtin-types.h. With multiple
schemata you now have multiple qapi-types.[ch], but only one
qapi-builtin-types.[ch]. Same for qapi-visit.[ch] and
qapi-builtin-visit.[ch].
Bonus: if all you need is built-in stuff, you can include a much
smaller header. To be exploited shortly.
Backports commit cdb6610ae4283720037bae2af1f78bd40eb5fe71 from qemu
Whenever qapi-schema.json changes, we run six programs eleven times to
update eleven files. Similar for qga/qapi-schema.json. This is
silly. Replace the six programs by a single program that spits out
all eleven files.
The programs become modules in new Python package qapi, along with the
helper library. This requires moving them to scripts/qapi/. While
moving them, consistently drop executable mode bits.
Backports commit fb0bc835e56b894cbc7236294921e5393c786ad8 from qemu
This reverts commit 42a77f1ce4934b243df003f95bda88530631387a.
The primary intention of this change was to silence messages
like
make[1]: '/home/berrange/src/virt/qemu/capstone/libcapstone.a' is up to date.
which we get when calling make recursively with explicit
targets.
The problem is that this change affected every make target,
not merely the targets that triggered these "is up to date"
messages. As a result any targets that were not invoking
commands via "$(call quiet-command ...)" suddenly become
silent. This is particularly bad for "make install" which
now appears todo nothing.
Rather than go through every make rule and try to identify
places where we now need to explicitly print a message to
show work taking place, just revert the change.
To address the original problem of silencing "is up to date"
messages, we simply add --quiet to the SUBDIR_MAKEVARS
variable, so it only affects us on recursive make calls.
Backports commit 8cc357b5a8dfba8ed11d1ce376afbc4ea35677a9 from qemu
Python2 did not validate locale correctness when reading input data, so
would happily read UTF-8 data in non-UTF-8 locales. Python3 is strict so
if you try to read UTF-8 data in the C locale, it will raise an error
for any UTF-8 bytes that aren't representable in 7-bit ascii encoding.
e.g.
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 54: ordinal not in range(128)
Traceback (most recent call last):
File "/tmp/qemu-test/src/scripts/qapi-commands.py", line 317, in <module>
schema = QAPISchema(input_file)
File "/tmp/qemu-test/src/scripts/qapi.py", line 1468, in __init__
parser = QAPISchemaParser(open(fname, 'r'))
File "/tmp/qemu-test/src/scripts/qapi.py", line 301, in __init__
previously_included)
File "/tmp/qemu-test/src/scripts/qapi.py", line 348, in _include
exprs_include = QAPISchemaParser(fobj, previously_included, info)
File "/tmp/qemu-test/src/scripts/qapi.py", line 271, in __init__
self.src = fp.read()
File "/usr/lib64/python3.5/encodings/ascii.py", line 26, in decode
return codecs.ascii_decode(input, self.errors)[0]
More background on this can be seen in
https://www.python.org/dev/peps/pep-0538/
Many distros support a new C.UTF-8 locale that is like the C locale,
but with UTF-8 instead of 7-bit ASCII. That is not entirely portable
though. This patch thus sets the LANG to "C", but overrides LC_CTYPE
to be en_US.UTF-8 locale. This gets us pretty close to C.UTF-8, but
in a way that should be portable to everywhere QEMU builds.
This patch only forces UTF-8 for QAPI scripts, since that is the one
showing the immediate error under Python3 with C locale, but potentially
we ought to force this for all python scripts used in the build process.
Backports commit d4e5ec877ca698a87dabe68814c6f93668f50c60 from qemu
Move generic make flags in MAKEFLAGS (SUBDIR_MAKEFLAGS is more qemu specific).
Use --quiet to silence make 'is up to date' message.
Backports commit 42a77f1ce4934b243df003f95bda88530631387a from qemu
For some systems (i.e. FreeBSD) the default 'make' is not compatible with the
GNU extensions used by QEMU makefiles.
Calling the GNU make (gmake) works, however the help displayed refers to the
host 'make' and copy/paste leads to lot of unobvious errors:
$ gmake check-help
[...]
make check Run all tests
$ make check
make: "Makefile" line 28: Missing dependency operator
make: "Makefile" line 37: Need an operator
make: "Makefile" line 41: warning: duplicate script for target "git-submodule-update" ignored
make: "rules.mak" line 70: warning: duplicate script for target "%.o" ignored
make: Unknown modifier ' '
make: Unclosed substitution for eval modules (= missing)
make: "tests/Makefile.include" line 24: Variable/Value missing from "export"
make: "tests/" line 1: warning: Zero byte read from file, skipping rest of line.
make: "tests/" line 1: Need an operator
make: "Makefile" line 660: warning: duplicate script for target "ifneq" ignored
make: "Makefile" line 78: warning: using previous script for "ifneq" defined here
make: Fatal errors encountered -- cannot continue
Using the $(MAKE) variable, the help displayed is consistent with the 'make'
program used.
Backports commit b98a3bae2596fd9bf60f140d042c8e993daba930 from qemu
Mandatory option is silly, and the error handling is missing: the
programs crash when -i isn't supplied. Make it an argument, and check
it properly.
Backports commit 16d80f61814745bd3f5bb9f47ae3b00edf9e1e45 from qemu
Future patches will be adding more crypto related APIs which
rely on QOM infrastructure. This creates a problem, because
QOM relies on library constructors to register objects. When
you have a file in a static .a library though which is only
referenced by a constructor the linker is dumb and will drop
that file when linking to the final executable :-( The only
workaround for this is to link the .a library to the executable
using the -Wl,--whole-archive flag, but this creates its own
set of problems because QEMU is relying on lazy linking for
libqemuutil.a. Using --whole-archive majorly increases the
size of final executables as they now contain a bunch of
object code they don't actually use.
The least bad option is to thus not include the crypto objects
in libqemuutil.la, and instead define a crypto-obj-y variable
that is referenced directly by all the executables that need
this code (tools + softmmu, but not qemu-ga). We avoid pulling
entire of crypto-obj-y into the userspace emulators as that
would force them to link to gnutls too, which is not required.
Backports commit fb37726db77b21f3731b90693d2c93ade1777528 from qemu
