Python 2 happily reads UTF-8 files in text mode, but Python 3 requires
either UTF-8 locale or an explicit encoding passed to open(). Commit
d4e5ec877ca fixed this by setting the en_US.UTF-8 locale. Falls apart
when the locale isn't be available.
Matthias Maier and Arfrever Frehtes Taifersar Arahesis proposed to use
binary mode instead, with manual conversion from bytes to str. Works,
but opening with an explicit encoding is simpler, so do that.
Since Python 2's open() doesn't support the encoding parameter, we
need to suppress it with a version check.
Backports commit de685ae5e9a4b523513033bd6cadc8187a227170 from qemu
It often happens that just a few discriminator values imply extra data in
a flat union. Existing checks did not make possible to leave other values
uncovered. Such cases had to be worked around by either stating a dummy
(empty) type or introducing another (subset) discriminator enumeration.
Both options create redundant entities in qapi files for little profit.
With this patch it is not necessary anymore to add designated union
fields for every possible value of a discriminator enumeration.
Backports commit 800877bb1639d38ffaebe312a37b61c66bb10c83 from qemu
The event generator produces an enum, and put it in the last visited
module. It fits better in the main module, since it's the set of all
visited events, from all modules.
Backports commit f030ffd39d6c1ea8fff281be5e4b19c819d7ce10 from qemu
To be more accurate on its purpose and make code that looks for a certain
target out of this variable more readable.
Backports commit 208ecb3e1acc8d55dab49fdf721a86d513691688 from qemu
Including only 4, as-yet unimplemented, instruction patterns
so that the whole thing compiles.
Backports commit 38388f7ee3adc04a7e7246c04352451c4f8d00fb from qemu
Now that we can safely call QOBJECT() on QObject * as well as its
subtypes, we can have macros qobject_ref() / qobject_unref() that work
everywhere instead of having to use QINCREF() / QDECREF() for QObject
and qobject_incref() / qobject_decref() for its subtypes.
The replacement is mechanical, except I broke a long line, and added a
cast in monitor_qmp_cleanup_req_queue_locked(). Unlike
qobject_decref(), qobject_unref() doesn't accept void *.
Note that the new macros evaluate their argument exactly once, thus no
need to shout them.
Backports commit cb3e7f08aeaab0ab13e629ce8496dca150a449ba from qemu
Move qapi-schema.json to qapi/, so it's next to its modules, and all
files get generated to qapi/, not just the ones generated for modules.
Consistently name the generated files qapi-MODULE.EXT:
qmp-commands.[ch] become qapi-commands.[ch], qapi-event.[ch] become
qapi-events.[ch], and qmp-introspect.[ch] become qapi-introspect.[ch].
This gets rid of the temporary hacks in scripts/qapi/commands.py,
scripts/qapi/events.py, and scripts/qapi/common.py.
Backports commit eb815e248f50cde9ab86eddd57eca5019b71ca78 from qemu
In my "build everything" tree, a change to the types in
qapi-schema.json triggers a recompile of about 4800 out of 5100
objects.
The previous commit split up qmp-commands.h, qmp-event.h, qmp-visit.h,
qapi-types.h. Each of these headers still includes all its shards.
Reduce compile time by including just the shards we actually need.
To illustrate the benefits: adding a type to qapi/migration.json now
recompiles some 2300 instead of 4800 objects. The next commit will
improve it further.
Backports commit 9af2398977a78d37bf184d6ff6bd04c72bfbf006 from qemu
Our qapi-schema.json is composed of modules connected by include
directives, but the generated code is monolithic all the same: one
qapi-types.h with all the types, one qapi-visit.h with all the
visitors, and so forth. These monolithic headers get included all
over the place. In my "build everything" tree, adding a QAPI type
recompiles about 4800 out of 5100 objects.
We wouldn't write such monolithic headers by hand. It stands to
reason that we shouldn't generate them, either.
Split up generated qapi-types.h to mirror the schema's modular
structure: one header per module. Name the main module's header
qapi-types.h, and sub-module D/B.json's header D/qapi-types-B.h.
Mirror the schema's includes in the headers, so that qapi-types.h gets
you everything exactly as before. If you need less, you can include
one or more of the sub-module headers. To be exploited shortly.
Split up qapi-types.c, qapi-visit.h, qapi-visit.c, qmp-commands.h,
qmp-commands.c, qapi-event.h, qapi-event.c the same way.
qmp-introspect.h, qmp-introspect.c and qapi.texi remain monolithic.
The split of qmp-commands.c duplicates static helper function
qmp_marshal_output_str() in qapi-commands-char.c and
qapi-commands-misc.c. This happens when commands returning the same
type occur in multiple modules. Not worth avoiding.
Since I'm going to rename qapi-event.[ch] to qapi-events.[ch], and
qmp-commands.[ch] to qapi-commands.[ch], name the shards that way
already, to reduce churn. This requires temporary hacks in
commands.py and events.py. Similarly, c_name() must temporarily
be taught to munge '/' in common.py. They'll go away with the rename.
Backports commit 252dc3105fc494182e236e97fe20f2d6b1d652cb from qemu
guardname() fails to return a valid C identifier for arguments
containing anything but [A-Za-z0-9_.-']. Fix that. Don't bother
protecting ticklish identifiers; header guards are all-caps, and no
ticklish identifiers are.
Backports commit f9c146399dabefb8cd13c9c467a9e710af15ea70 from qemu
Linking code from multiple separate QAPI schemata into the same
program is possible, but involves some weirdness around built-in
types:
* We generate code for built-in types into .c only with option
--builtins. The user is responsible for generating code for exactly
one QAPI schema per program with --builtins.
* We generate code for built-in types into .h regardless of
--builtins, but guarded by #ifndef QAPI_VISIT_BUILTIN. Because all
copies of this code are exactly the same, including any combination
of these headers works.
Replace this contraption by something more conventional: generate code
for built-in types into their very own files: qapi-builtin-types.c,
qapi-builtin-visit.c, qapi-builtin-types.h, qapi-builtin-visit.h, but
only with --builtins. Obey --output-dir, but ignore --prefix for
them.
Make qapi-types.h include qapi-builtin-types.h. With multiple
schemata you now have multiple qapi-types.[ch], but only one
qapi-builtin-types.[ch]. Same for qapi-visit.[ch] and
qapi-builtin-visit.[ch].
Bonus: if all you need is built-in stuff, you can include a much
smaller header. To be exploited shortly.
Backports commit cdb6610ae4283720037bae2af1f78bd40eb5fe71 from qemu
The use of QAPIGen is rather shallow so far: most of the output
accumulation is not converted. Take the next step: convert output
accumulation in the code-generating visitor classes. Helper functions
outside these classes are not converted.
Backports commit 71b3f0459c460c9e16a47372ccddbfa6e2c7aadf from qemu
The include directive permits modular QAPI schemata, but the generated
code is monolithic all the same. To permit generating modular code,
the front end needs to pass more information on inclusions to the back
ends. The commit before last added the necessary information to the
parse tree. This commit adds it to the intermediate representation
and its QAPISchemaVisitor. A later commit will use this to to
generate modular code.
New entity QAPISchemaInclude represents inclusions. Call new visitor
method visit_include() for it, so visitors can see the sub-modules a
module includes.
Note that unlike other entities, QAPISchemaInclude has no name, and is
therefore not added to entity_dict.
New QAPISchemaEntity attribute @module names the entity's source file.
Call new visitor method visit_module() when it changes during a visit,
so visitors can keep track of the module being visited.
Backports commit cf40a0a5c2e1091846974cc8cc95a60e0b1db4af from qemu
The generators' conversion to visitors (merge commit 9e72681d16)
changed the processing order of entities from source order to
alphabetical order. The next commit needs source order, so change it
back.
Backports commit 8a84767cc4f7e00e5dd62435c32be9e7d2cbe4d3 from qemu
The parse tree is a list of expressions. Except include expressions
currently get replaced by the included file's parse tree.
Instead of throwing away the include expression, keep it with the file
name expanded so you don't have to track the including file's
directory to make sense of it.
A future commit will put this include expression to use.
Backports commit 97f0249474d19c1d60fb9d934c8bc08625a619ca from qemu
Error messages print absolute file names of included files even if the
user gave a relative one on the command line:
$ PYTHONPATH=scripts python -B tests/qapi-schema/test-qapi.py tests/qapi-schema/include-cycle.json
In file included from tests/qapi-schema/include-cycle.json:1:
In file included from /work/armbru/qemu/tests/qapi-schema/include-cycle-b.json:1:
/work/armbru/qemu/tests/qapi-schema/include-cycle-c.json:1: Inclusion loop for include-cycle.json
Improve this to
In file included from tests/qapi-schema/include-cycle.json:1:
In file included from tests/qapi-schema/include-cycle-b.json:1:
tests/qapi-schema/include-cycle-c.json:1: Inclusion loop for include-cycle.json
The error message when an include file can't be opened prints the
include directive's file name, which is relative to the including
file. Change this to print the file name relative to the working
directory. Visible in tests/qapi-schema/include-no-file.err.
Backports commit af97502ce9c648ae5c746b9e562d6e4586f02eee from qemu
A massive number of objects depends on QAPI-generated headers. In my
"build everything" tree, it's roughly 4800 out of 5100. This is
particularly annoying when only some of the generated files change,
say for a doc fix.
Improve qapi-gen.py to touch its output files only if they actually
change. Rebuild time for a QAPI doc fix drops from many minutes to a
few seconds. Rebuilds get faster for certain code changes, too. For
instance, adding a simple QMP event now recompiles less than 200
instead of 4800 objects. But adding a QAPI type is as bad as ever;
we've clearly got more work to do.
Backports commit 907b846653fb3757bf2ab98d6d66f92df34d875f from qemu
Whenever qapi-schema.json changes, we run six programs eleven times to
update eleven files. Similar for qga/qapi-schema.json. This is
silly. Replace the six programs by a single program that spits out
all eleven files.
The programs become modules in new Python package qapi, along with the
helper library. This requires moving them to scripts/qapi/. While
moving them, consistently drop executable mode bits.
Backports commit fb0bc835e56b894cbc7236294921e5393c786ad8 from qemu
The next commit will introduce a common driver program for all
generators. The generators need to be modules for that. qapi2texi.py
already is. Make the other generators follow suit.
The changes are actually trivial. Obvious in the diffs once you view
them with whitespace changes ignored.
Backports commit 26df4e7fab06422b21e11d039c64243ca4003147 from qemu
In preparation of the next commit, which will turn the generators into
modules. These global variables will become local to main() then.
Backports commit 93b564c444edc41901d0f7e922833eeb751f8249 from qemu
These classes encapsulate accumulating and writing output.
Convert C code generation to QAPIGenC and QAPIGenH. The conversion is
rather shallow: most of the output accumulation is not converted.
Left for later.
The indentation machinery uses a single global variable indent_level,
even though we generally interleave creation of a .c and its .h. It
should become instance variable of QAPIGenC. Also left for later.
Documentation generation isn't converted, and QAPIGenDoc isn't used.
This will change shortly.
Backports commit 47a6ea9aab1d857015684cda387ffba05a036721 from qemu
Rename the variable holding the QAPISchemaGenFOOVisitor from gen to
vis, to avoid confusion in the next commit.
Backports commit d46eec4260540d83bafba91608842ab03dabf339 from qemu
Each generator carries a copyright notice for the generator itself,
and another one for the files it generates. Only the former have been
updated along the way, the latter have not, and are all out of date.
Fix by copying the generator's copyright notice to the generated files
instead. Note that the fix doesn't copy the "Authors:" part; the
generated files' outdated Authors list goes away without replacement.
Backports commit 5ddeec83eb0284b52bb3d496a49ba1657069ed45 from qemu
Every generator has separate boilerplate for .h and .c, and their
differences are boring. All of them repeat the license note.
Reduce the repetition as follows. Move common text like the license
note to common open_output(), next to the existing common text there.
For each generator, replace the two separate descriptions by a single
one.
While there, emit an "automatically generated" note into generated
documentation, too.
Backports commit c263de3f419be945499ff7e6bd7512702f8bd522 from qemu
This cleanup makes the number of objects depending on qapi/qmp/qdict.h
drop from 4550 (out of 4743) to 368 in my "build everything" tree.
For qapi/qmp/qobject.h, the number drops from 4552 to 390.
While there, separate #include from file comment with a blank line.
Backports commit 452fcdbc49c59884c8c284268d64baa24fea11e1 from qemu
The conflict check added by commit c0644771 ("qapi: Reject
alternates that can't work with keyval_parse()") doesn't work
with the following declaration:
{ 'alternate': 'Alt',
'data': { 'one': 'bool',
'two': 'str' } }
It crashes with:
Traceback (most recent call last):
File "./scripts/qapi-types.py", line 295, in <module>
schema = QAPISchema(input_file)
File "/home/ehabkost/rh/proj/virt/qemu/scripts/qapi.py", line 1468, in __init__
self.exprs = check_exprs(parser.exprs)
File "/home/ehabkost/rh/proj/virt/qemu/scripts/qapi.py", line 958, in check_exprs
check_alternate(expr, info)
File "/home/ehabkost/rh/proj/virt/qemu/scripts/qapi.py", line 830, in check_alternate
% (name, key, types_seen[qtype]))
KeyError: 'QTYPE_QSTRING'
This happens because the previously-seen conflicting member
('one') can't be found at types_seen[qtype], but at
types_seen['QTYPE_BOOL'].
Fix the bug by moving the error check to the same loop that adds
new items to types_seen, raising an exception if types_seen[qt]
is already set.
Backports commit fda72ab4510bcc680a3c4fe55997aa29589884f7 from qemu
This cleanup makes the number of objects depending on qapi/error.h
drop from 1910 (out of 4743) to 1612 in my "build everything" tree.
While there, separate #include from file comment with a blank line,
and drop a useless comment on why qemu/osdep.h is included first.
Backports commit e688df6bc4549f28534cdb001f168b8caae55b0c from qemu
Some early python 3.x versions will have different default
ordering when calling the 'values()' method on a dict, compared
to python 2.x and later 3.x versions. Explicitly sort the items
to get a stable ordering.
Backports commit f7a5376d4b667cf6c83c1d640e32d22456d7b5ee from qemu
The OrderedDict class appeared in the 'collections' module
from python 2.7 onwards, so use that in preference to our
local backport if available.
Backports commit 38710a8994911d98acbe183a39ec3a53638de510 from qemu
The iteritems()/itervalues() methods are gone in py3, but the
items()/values() methods are still around. The latter are less
efficient than the former in py2, but this has unmeasurably
small impact on QEMU build time, so taking portability over
efficiency is a net win.
Backports commit 2f8480447067d6f42af52a886385284ead052af9 from qemu
Python 3 no longer supports the bare "print" statement, it must be
called as a normal function with round brackets. It is possible to
opt-in to this new syntax with Python 2.6 onwards by importing the
"print_function" from the "__future__" module, making it easy to
support Python 2 and 3 in parallel.
Backports commit ef9d9108917d6d5f903bca31602827e512a51c50 from qemu
The gen_ prefix is awkward. Generated C should go through cgen()
exactly once (see commit 1f9a7a1). The common way to get this wrong is
passing a foo=gen_foo() keyword argument to mcgen(). I'd like us to
adopt a naming convention where gen_ means "something that's been piped
through cgen(), and thus must not be passed to cgen() or mcgen()".
Requires renaming gen_params(), gen_marshal_proto() and
gen_event_send_proto().
Backports commit 086ee7a6200fa5ad795b12110b5b3d5a93dcac3e from qemu
Before the previous commit, parameter promote_int = true made
visit_start_alternate() with an input visitor avoid QTYPE_QINT
variants and create QTYPE_QFLOAT variants instead. This was used
where QTYPE_QINT variants were invalid.
The previous commit fused QTYPE_QINT with QTYPE_QFLOAT, rendering
promote_int useless and unused.
Backports commit 60390d2dc85ffade8981ca41e02335cb07353a6d from qemu
We would like to use a same QObject type to represent numbers, whether
they are int, uint, or floats. Getters will allow some compatibility
between the various types if the number fits other representations.
Add a few more tests while at it.
Backports commit 01b2ffcedd94ad7b42bc870e4c6936c87ad03429 from qemu
The QmpOutputVisitor has no direct dependency on QMP. It is
valid to use it anywhere that one wants a QObject. Rename it
to better reflect its functionality as a generic QAPI
to QObject converter.
The commit before previous renamed the files, this one renames C
identifiers.
Backports commit 7d5e199ade76c53ec316ab6779800581bb47c50a from qemu
The QMP visitors have no direct dependency on QMP. It is
valid to use them anywhere that one has a QObject. Rename them
to better reflect their functionality as a generic QObject
to QAPI converter.
This is the first of three parts: rename the files. The next two
parts will rename C identifiers. The split is necessary to make git
rename detection work.
Backports commit b3db211f3c80bb996a704d665fe275619f728bd4 from qemu
Turn on the ability to pass command and event arguments in
a single boxed parameter, which must name a non-empty type
(although the type can be a struct with all optional members).
For structs, it makes it possible to pass a single qapi type
instead of a breakout of all struct members (useful if the
arguments are already in a struct or if the number of members
is large); for other complex types, it is now possible to use
a union or alternate as the data for a command or event.
The empty type may be technically feasible if needed down the
road, but it's easier to forbid it now and relax things to allow
it later, than it is to allow it now and have to special case
how the generated 'q_empty' type is handled (see commit 7ce106a9
for reasons why nothing is generated for the empty type). An
alternate type is never considered empty, but now that a boxed
type can be either an object or an alternate, we have to provide
a trivial QAPISchemaAlternateType.is_empty(). The new call to
arg_type.is_empty() during QAPISchemaCommand.check() requires
that we first check the type in question; but there is no chance
of introducing a cycle since objects do not refer back to commands.
We still have a split in syntax checking between ad-hoc parsing
up front (merely validates that 'boxed' has a sane value) and
during .check() methods (if 'boxed' is set, then 'data' must name
a non-empty user-defined type).
Generated code is unchanged, as long as no client uses the
new feature.
Backports commit c818408e449ea55371253bd4def1c1dc87b7bb03 from qemu
The next patch will add support for passing a qapi union type
as the 'data' of a command. But to do that, the user function
for implementing the command, as called by the generated
marshal command, must take the corresponding C struct as a
single boxed pointer, rather than a breakdown into one
parameter per member. Even without a union, being able to use
a C struct rather than a list of parameters can make it much
easier to handle coding with QAPI.
This patch adds the internal plumbing of a 'boxed' flag
associated with each command and event. In several cases,
this means adding indentation, with one new dead branch and
the remaining branch being the original code more deeply
nested; this was done so that the new implementation in the
next patch is easier to review without also being mixed with
indentation changes.
For this patch, no behavior or generated output changes, other
than the testsuite outputting the value of the new flag
(always False for now).
Backports commit 48825ca419fd9c8140d4fecb24e982d68ebca74f from qemu
Commit 7ce106a9 documented why we don't generated a visit_type_FOO()
for implicit types; and therefore events with an anonymous type for
'data' have to open-code a visit. Note that the open-coded visit in
qapi-event.c is slightly different from what is done in
qapi-visit.c for normal types, in part because we don't have to
check for *obj being NULL or free things on error. But where the
type is not implicit, it is nicer to reuse the normal visit instead
of open-coding a duplicate.
At the moment, the only event with a non-implicit 'data' is in the
testsuite, where test-qapi-event.c changes as follows:
|@@ -155,6 +155,7 @@ void qapi_event_send___org_qemu_x_event(
| __org_qemu_x_Struct param = {
| __org_qemu_x_member1, (char *)__org_qemu_x_member2, has_q_wchar_t, q_wchar_t
| };
|+ __org_qemu_x_Struct *arg = ¶m;
|
| emit = qmp_event_get_func_emit();
| if (!emit) {
|@@ -164,16 +165,7 @@ void qapi_event_send___org_qemu_x_event(
| qmp = qmp_event_build_dict("__ORG.QEMU_X-EVENT");
|
| v = qmp_output_visitor_new(&obj);
|-
|- visit_start_struct(v, "__ORG.QEMU_X-EVENT", NULL, 0, &err);
|- if (err) {
|- goto out;
|- }
|- visit_type___org_qemu_x_Struct_members(v, ¶m, &err);
|- if (!err) {
|- if (!err) {
|- visit_check_struct(v, &err);
|- }
|- visit_end_struct(v, NULL);
|+ visit_type___org_qemu_x_Struct(v, "__ORG.QEMU_X-EVENT", &arg, &err);
| if (err) {
| goto out;
| }
Backports commit 4d0b268fdb17a1fed10fe980e77fd388e5427bfd from qemu
Ever since commit 12f254f removed the last parameterization
of gen_err_check(), it no longer makes sense to hide the three
lines of generated C code behind a macro call. Just inline it
into the remaining users.
No change to generated code.
Backports commit fa274ed6fb788866ed3a2cfd54a2ddf78f04f2c0 from qemu
In the near future, we want to lift our artificial restriction of
no variants at the top level of an event, at which point the
currently open-coded check for empty members will become
insufficient. Factor it out into a new helper method is_empty()
now, and future-proof it by checking variants, too, along with an
assert that it is not used prior to the completion of .check().
Update places that were checking for (non-)empty .members to use
the new helper.
All of the current callers assert that there are no variants (either
directly, or by qapi.py asserting that base types have no variants),
so this is not a semantic change.
No change to generated code.
Backports commit b6167706829c6e0d3572daa2b6769594ced276f7 from qemu
