Commit graph

2818 commits

Author SHA1 Message Date
Eric Blake b9b76f9b4b
qapi: Lazy creation of array types
Commit ac88219a had several TODO markers about whether we needed
to automatically create the corresponding array type alongside
any other type. It turns out that most of the time, we don't!

There are a few exceptions: 1) We have a few situations where we
use an array type in internal code but do not expose that type
through QMP; fix it by declaring a dummy type that forces the
generator to see that we want to use the array type.

2) The builtin arrays (such as intList for QAPI ['int']) must
always be generated, because of the way our QAPI_TYPES_BUILTIN
compile guard works: we have situations (at the very least
tests/test-qmp-output-visitor.c) that include both top-level
"qapi-types.h" (via "error.h") and a secondary
"test-qapi-types.h". If we were to only emit the builtin types
when used locally, then the first .h file would not include all
types, but the second .h does not declare anything at all because
the first .h set QAPI_TYPES_BUILTIN, and we would end up with
compilation error due to things like unknown type 'int8List'.

Actually, we may need to revisit how we do type guards, and
change from a single QAPI_TYPES_BUILTIN over to a different
usage pattern that does one #ifdef per qapi type - right now,
the only types that are declared multiple times between two qapi
.json files for inclusion by a single .c file happen to be the
builtin arrays. But now that we have QAPI 'include' statements,
it is logical to assume that we will soon reach a point where
we want to reuse non-builtin types (yes, I'm thinking about what
it will take to add introspection to QGA, where we will want to
reuse the SchemaInfo type and friends). One #ifdef per type
will help ensure that generating the same qapi type into more
than one qapi-types.h won't cause collisions when both are
included in the same .c file; but we also have to solve how to
avoid creating duplicate qapi-types.c entry points. So that
is a problem left for another day.

Generated code for qapi-types and qapi-visit is drastically
reduced; less than a third of the arrays that were blindly
created were actually needed (a quick grep shows we dropped
from 219 to 69 *List types), and the .o files lost more than
30% of their bulk. [For best results, diff the generated
files with 'git diff --patience --no-index pre post'.]

Interestingly, the introspection output is unchanged - this is
because we already cull all types that are not indirectly
reachable from a command or event, so introspection was already
using only a subset of array types. The subset of types
introspected is now a much larger percentage of the overall set
of array types emitted in qapi-types.h (since the larger set
shrunk), but still not 100% (evidence that the array types
emitted for our new Dummy structs, and the new struct itself,
don't affect QMP).

Backports commit 9f08c8ec73878122ad4b061ed334f0437afaaa32 from qemu
2018-02-19 18:55:35 -05:00
Eric Blake dc65960e1f
qapi: Don't use info as witness of implicit object type
A future patch will enable error reporting from the various
QAPISchema*.check() methods. But to report an error related
to an implicit type, we'll need to associate a location with
the type (the same location as the top-level entity that is
causing the creation of the implicit type), and once we do
that, keying off of whether foo.info exists is no longer a
viable way to determine if foo is an implicit type.

Instead, add an is_implicit() method to QAPISchemaEntity, and use it.
It can be overridden later for ObjectType and EnumType, when implicit
instances of those classes gain info.

Backports commit 49823c4b4304a3e4aa5d67e089946b12d6a52d64 from qemu
2018-02-19 18:53:06 -05:00
Eric Blake 3d11f7173d
qapi: Prepare for errors during check()
The next few patches will start migrating error checking from
ad hoc parse methods into the QAPISchema*.check() methods. But
for an error message to display, we first have to fix the
overall 'try' to catch those errors. We also want to enable a
few more assertions, such as making sure every attempt to
raise a semantic error is passed a valid location info, or that
various preconditions hold.

The general approach for moving error checking will then be to
relax an assertion into an if that raises an exception if the
condition does not hold, and removing the counterpart ad hoc
check done during the parse phase.

Backports commit 7618b91ff80ec42b84b29be24d8ef53ddb377110 from qemu
2018-02-19 18:50:23 -05:00
Eric Blake 9d0c5746ff
qapi: Use predicate callback to determine visit filtering
Previously, qapi-types and qapi-visit filtered out implicit
objects during visit_object_type() by using 'info' (works since
implicit objects do not [yet] have associated info); meanwhile
qapi-introspect filtered out all schema types on the first pass
by returning a python type from visit_begin(), which was then
used at a distance in QAPISchema.visit() to do the filtering.

Rather than keeping these ad hoc approaches, add a new visitor
callback visit_needed() which returns False to skip a given
entity, and which defaults to True unless overridden. Use the
new mechanism to simplify all three filtering visitors.

No change to the generated code.

Backports commit 25a0d9c977c2f5db914b0a1619759fd77d97b016 from qemu
2018-02-19 18:49:08 -05:00
Eric Blake f371fa9105
qapi: Simplify gen_visit_fields() error handling
Since we have consolidated all generated code to use 'err' as
the name of the local variable for error detection, we can
simplify the decision on whether to skip error detection (useful
for deallocation paths) to be a boolean.

Backports commit 18bdbc3ac8b477e160d56aa6ecd6942495ce44d0 from qemu
2018-02-19 18:45:05 -05:00
Eric Blake 4ca10929a7
qapi: Share gen_visit_fields()
Consolidate the code between visit, command marshalling, and
event generation that iterates over the members of a struct.
It reduces code duplication in the generator, so that a future
patch can reduce the size of generated code while touching only
one instead of three locations.

There are no changes to the generated marshal code.

The visitor code becomes slightly more verbose, but remains
semantically equivalent, and is actually easier to read as
it follows a more common idiom:

| visit_optional(v, &(*obj)->has_device, "device", &err);
|- if (!err && (*obj)->has_device) {
|- visit_type_str(v, &(*obj)->device, "device", &err);
|- }
| if (err) {
| goto out;
| }
|+ if ((*obj)->has_device) {
|+ visit_type_str(v, &(*obj)->device, "device", &err);
|+ if (err) {
|+ goto out;
|+ }
|+ }

The event code becomes slightly more verbose, but this is
arguably a bug fix: although the visitors are not well
documented, use of an optional member should not be attempted
unless guarded by a prior call to visit_optional(). Works only
because the output qmp visitor has a no-op visit_optional():

|+ visit_optional(v, &has_offset, "offset", &err);
|+ if (err) {
|+ goto out;
|+ }
| if (has_offset) {
| visit_type_int(v, &offset, "offset", &err);

Backports commit 82ca8e469666b169ccf818a0e36136aee97d7db0 from qemu
2018-02-19 18:41:41 -05:00
Eric Blake 13b3a6c92f
qapi: Consistent generated code: minimize push_indent() usage
We had some pointless differences in the generated code for visit,
command marshalling, and events; unifying them makes it easier for
future patches to consolidate to common helper functions.
This is one patch of a series to clean up these differences.

This patch reduces the number of push_indent()/pop_indent() pairs
so that generated code is typically already at its natural output
indentation in the python files. It is easier to reason about
generated code if the reader does not have to track how much
spacing will be inserted alongside the code, and moreso when all
of the generators use the same patterns (qapi-type and qapi-event
were already using in-place indentation).

Arguably, the resulting python may be a bit harder to read with C
code at the same indentation as python; on the other hand, not
having to think about push_indent() is a win, and most decent
editors provide syntax highlighting that makes it easier to
visually distinguish python code from string literals that will
become C code.

There is no change to the generated output.

Backports commit 05372f708a8cb3556e4d67458de79417dadf241f from qemu
2018-02-19 18:38:48 -05:00
Eric Blake ec89e19b30
qapi: Consistent generated code: prefer common indentation
We had some pointless differences in the generated code for visit,
command marshalling, and events; unifying them makes it easier for
future patches to consolidate to common helper functions.
This is one patch of a series to clean up these differences.

This patch adjusts gen_visit_union() to use the same indentation
as other functions, namely, by jumping early to the error label
if the object was not set rather than placing the rest of the
body inside an if for when it is set.

No change in semantics to the generated code.

Backports commit e36c714e6aad7c9266132350833e2f263f6d8874 from qemu
2018-02-19 18:26:14 -05:00
Eric Blake 53c2c709af
qapi: Consistent generated code: prefer common labels
We had some pointless differences in the generated code for visit,
command marshalling, and events; unifying them makes it easier for
future patches to consolidate to common helper functions.
This is one patch of a series to clean up these differences.

This patch names the goto labels 'out' (not 'clean') and 'out_obj'
(not 'out_end'). Additionally, the generator was inconsistent on
whether labels had a leading space [our HACKING is silent; while
emacs 'gnu' style adds the space to avoid littering column 1].
For minimal churn, prefer no leading space; this also matches
the style that is more prevalent in current qemu.git.

No change in semantics to the generated code.

Backports commit f782399cb4fa3fc4182cb046817f65a6db92ab07 from qemu
2018-02-19 18:23:15 -05:00
Eric Blake 4e9c5e4d52
qapi: Consistent generated code: prefer visitor 'v'
We had some pointless differences in the generated code for visit,
command marshalling, and events; unifying them makes it easier for
future patches to consolidate to common helper functions.
This is one patch of a series to clean up these differences.

This patch names the local visitor variable 'v' rather than 'm'.
Related objects, such as 'QapiDeallocVisitor', are also named by
their initials instead of an unrelated leading m.

No change in semantics to the generated code.

Backports commit f8b7f1a8eafa9f565ebecfe409e8741d38cd786b from qemu
2018-02-19 18:21:53 -05:00
Eric Blake bc8a7ae5f9
qapi: Consistent generated code: prefer error 'err'
We had some pointless differences in the generated code for visit,
command marshalling, and events; unifying them makes it easier for
future patches to consolidate to common helper functions.
This is one patch of a series to clean up these differences.

This patch consistently names the local error variable 'err' rather
than 'local_err'.

No change in semantics to the generated code.

Backports commit 2a0f50e8d973b01eda4c63bac4a5c79ea0f584ef from qemu
2018-02-19 18:15:10 -05:00
Eric Blake 4294815ee3
qapi: Reuse code for flat union base validation
Rather than open-code the check for a valid base type, we
should reuse the common functionality. This allows for
consistent error messages, and also makes it easier for a
later patch to turn on support for inline anonymous base
structures.

Test flat-union-inline is updated to test only one feature
(anonymous branch dictionaries), which can be implemented
independently (test flat-union-bad-base already covers the
idea of an anonymous base dictionary).

Backports commit 376863ef4895ae709aadb6f26365a5973310ef09 from qemu
2018-02-19 18:14:53 -05:00
Eric Blake c09db972fe
qapi: Avoid assertion failure on union 'type' collision
The previous commit added two tests that triggered an assertion
failure. It's fairly straightforward to avoid the failure by
just outright forbidding the collision between a union's tag
values and its discriminator name (including the implicit name
'kind' supplied for simple unions [*]). Ultimately, we'd like
to move the collision detection into QAPISchema*.check(), but
for now it is easier just to enhance the existing checks.

[*] Of course, down the road, we have plans to rename the simple
union tag name to 'type' to match the QMP wire name, but the
idea of the collision will still be present even then.

Technically, we could avoid the collision by naming the C union
members representing each enum value as '_case_value' rather
than 'value'; but until we have an actual qapi client (and not
just our testsuite) that has a legitimate reason to match a
case label to the name of a QMP key and needs the name munging
to satisfy the compiler, it's easier to just reject the qapi
as invalid.

Backports commit 7b2a5c2f9a52c4a08630fa741052f03fe5d3cc8a from qemu
2018-02-19 18:09:13 -05:00
Eric Blake 8fce54a867
qapi: Clean up qapi.py per pep8
Silence pep8, and make pylint a bit happier. Just style cleanups,
plus killing a useless comment in camel_to_upper(); no semantic
changes.

Backports commit 437db2549be383e52acad6cd4bf2862e98fdfc93 from qemu
2018-02-19 18:07:55 -05:00
Eric Blake 07f0fa3ee9
qapi: Invoke exception superclass initializer
pylint recommends that every exception class should explicitly
invoke the superclass __init__, even though things seem to work
fine without it.

Backports commit 59b00542659c8947f9d4e8c28d2d528ab3ab61a5 from qemu
2018-02-19 17:57:21 -05:00
Eric Blake de2b67e528
qapi: Improve 'include' error message
Use of '"...%s" % include' to print non-strings can lead to
ugly messages, such as this (if the .json change is applied
without the qapi.py change):
 Expected a file name (string), got: OrderedDict()

Better is to just omit the actual non-string value in the
message.

Backports commit 7408fb67c0f9403f6e40aecf97cf798fc14e2cd8 from qemu
2018-02-19 17:56:17 -05:00
Markus Armbruster 02e411f666
qapi: New QMP command query-qmp-schema for QMP introspection
qapi/introspect.json defines the introspection schema. It's designed
for QMP introspection, but should do for similar uses, such as QGA.

The introspection schema does not reflect all the rules and
restrictions that apply to QAPI schemata. A valid QAPI schema has an
introspection value conforming to the introspection schema, but the
converse is not true.

Introspection lowers away a number of schema details, and makes
implicit things explicit:

* The built-in types are declared with their JSON type.

All integer types are mapped to 'int', because how many bits we use
internally is an implementation detail. It could be pressed into
external interface service as very approximate range information,
but that's a bad idea. If we need range information, we better do
it properly.

* Implicit type definitions are made explicit, and given
auto-generated names:

- Array types, named by appending "List" to the name of their
element type, like in generated C.

- The enumeration types implicitly defined by simple union types,
named by appending "Kind" to the name of their simple union type,
like in generated C.

- Types that don't occur in generated C. Their names start with ':'
so they don't clash with the user's names.

* All type references are by name.

* The struct and union types are generalized into an object type.

* Base types are flattened.

* Commands take a single argument and return a single result.

Dictionary argument or list result is an implicit type definition.

The empty object type is used when a command takes no arguments or
produces no results.

The argument is always of object type, but the introspection schema
doesn't reflect that.

The 'gen': false directive is omitted as implementation detail.

The 'success-response' directive is omitted as well for now, even
though it's not an implementation detail, because it's not used by
QMP.

* Events carry a single data value.

Implicit type definition and empty object type use, just like for
commands.

The value is of object type, but the introspection schema doesn't
reflect that.

* Types not used by commands or events are omitted.

Indirect use counts as use.

* Optional members have a default, which can only be null right now

Instead of a mandatory "optional" flag, we have an optional default.
No default means mandatory, default null means optional without
default value. Non-null is available for optional with default
(possible future extension).

* Clients should *not* look up types by name, because type names are
not ABI. Look up the command or event you're interested in, then
follow the references.

TODO Should we hide the type names to eliminate the temptation?

New generator scripts/qapi-introspect.py computes an introspection
value for its input, and generates a C variable holding it.

It can generate awfully long lines. Marked TODO.

A new test-qmp-input-visitor test case feeds its result for both
tests/qapi-schema/qapi-schema-test.json and qapi-schema.json to a
QmpInputVisitor to verify it actually conforms to the schema.

New QMP command query-qmp-schema takes its return value from that
variable. Its reply is some 85KiBytes for me right now.

If this turns out to be too much, we have a couple of options:

* We can use shorter names in the JSON. Not the QMP style.

* Optionally return the sub-schema for commands and events given as
arguments.

Right now qmp_query_schema() sends the string literal computed by
qmp-introspect.py. To compute sub-schema at run time, we'd have to
duplicate parts of qapi-introspect.py in C. Unattractive.

* Let clients cache the output of query-qmp-schema.

It changes only on QEMU upgrades, i.e. rarely. Provide a command
query-qmp-schema-hash. Clients can have a cache indexed by hash,
and re-query the schema only when they don't have it cached. Even
simpler: put the hash in the QMP greeting.

Backports commit 39a181581650f4d50f4445bc6276d9716cece050 from qemu
2018-02-19 17:54:03 -05:00
Markus Armbruster 8115f998e9
qapi: Pseudo-type '**' is now unused, drop it
'gen': false needs to stay for now, because netdev_add is still using
it.

Backports commit 2d21291ae645955fcc4652ebfec81ad338169ac6 from qemu
2018-02-19 17:49:46 -05:00
Markus Armbruster e9dac6908f
qom: Don't use 'gen': false for qom-get, qom-set, object-add
With the previous commit, the generated marshalers just work, and save
us a bit of handwritten code.

Backports commit 6eb3937e9b20319e1c4f4d53e906fda8f5ccda10 from qemu
2018-02-19 17:47:59 -05:00
Markus Armbruster f93438ba43
qapi: Introduce a first class 'any' type
It's first class, because unlike '**', it actually works, i.e. doesn't
require 'gen': false.

'**' will go away next.

Backports commit 28770e057f265a4e70bcbdfc2447cce7b5f2dc19 from qemu
2018-02-19 17:46:58 -05:00
Markus Armbruster 66def00922
qapi: De-duplicate parameter list generation
Generated qapi-event.[ch] lose line breaks. No change otherwise.

Backports commit 03b4367a556179e3e59affa535493427bd009e9d from qemu
2018-02-19 17:38:52 -05:00
Markus Armbruster b4187c01da
qapi-visit: Rearrange code a bit
Move gen_visit_decl() to a better place. Inline
generate_visit_struct_body().

Backports commit 60f8546acd113e636bf2ba8af991ebe0f6e8ad66 from qemu
2018-02-19 17:37:09 -05:00
Markus Armbruster d6866a50f6
qapi: Clean up after recent conversions to QAPISchemaVisitor
Generate just 'FOO' instead of 'struct FOO' when possible.

Drop helper functions that are now unused.

Make pep8 and pylint reasonably happy.

Rename generate_FOO() functions to gen_FOO() for consistency.

Use more consistent and sensible variable names.

Consistently use c_ for mapping keys when their value is a C
identifier or type.

Simplify gen_enum() and gen_visit_union()

Consistently use single quotes for C text string literals.

Backports commit e98859a9b96d71dea8f9af43325edd43c7effe66 from qemu
2018-02-19 17:34:32 -05:00
Markus Armbruster 64e9fceab9
qapi: Replace dirty is_c_ptr() by method c_null()
is_c_ptr() looks whether the end of the C text for the type looks like
a pointer. Works, but is fragile.

We now have a better tool: use QAPISchemaType method c_null(). The
initializers for non-pointers become prettier: 0, false or the
enumeration constant with the value 0 instead of {0}.

Backports commit 5710153e7310995b5d4127af267e36d8529b3b30 from qemu
2018-02-19 17:17:40 -05:00
Markus Armbruster abfa5da7da
qapi-event: Convert to QAPISchemaVisitor, fixing data with base
Fixes events whose data is struct with base to include the struct's
base members. Test case is qapi-schema-test.json's event
__org.qemu_x-command:

{ 'event': '__ORG.QEMU_X-EVENT', 'data': '__org.qemu_x-Struct' }

{ 'struct': '__org.qemu_x-Struct', 'base': '__org.qemu_x-Base',
'data': { '__org.qemu_x-member2': 'str' } }

{ 'struct': '__org.qemu_x-Base',
'data': { '__org.qemu_x-member1': '__org.qemu_x-Enum' } }

Patch's effect on generated qapi_event_send___org_qemu_x_event():

-void qapi_event_send___org_qemu_x_event(const char *__org_qemu_x_member2,
+void qapi_event_send___org_qemu_x_event(__org_qemu_x_Enum __org_qemu_x_member1,
+ const char *__org_qemu_x_member2,
Error **errp)
{
QDict *qmp;
@@ -224,6 +225,10 @@ void qapi_event_send___org_qemu_x_event(
goto clean;
}

+ visit_type___org_qemu_x_Enum(v, &__org_qemu_x_member1, "__org.qemu_x-member1", &local_err);
+ if (local_err) {
+ goto clean;
+ }
visit_type_str(v, (char **)&__org_qemu_x_member2, "__org.qemu_x-member2", &local_err);
if (local_err) {
goto clean;

Code is generated in a different order now, but that doesn't matter.

Backports commit 05f43a960877cf941635324b2d0a74c0d0f7128e from qemu
2018-02-19 17:16:31 -05:00
Markus Armbruster 533bf08ff1
qapi-event: Eliminate global variable event_enum_value
Backports commit 7b24626cd019ed5084c8e3370999176a1ebd44be from qemu
2018-02-19 17:13:37 -05:00
Markus Armbruster 782a549898
qapi: De-duplicate enum code generation
Duplicated in commit 21cd70d. Yes, we can't import qapi-types, but
that's no excuse. Move the helpers from qapi-types.py to qapi.py, and
replace the duplicates in qapi-event.py.

The generated event enumeration type's lookup table becomes
const-correct (see commit 2e4450f), and uses explicit indexes instead
of relying on order (see commit 912ae9c).

Backports commit efd2eaa6c2992c214a13f102b6ddd4dca4697fb3 from qemu
2018-02-19 17:11:05 -05:00
Markus Armbruster baa36c6c88
qapi-visit: Convert to QAPISchemaVisitor, fixing bugs
Fixes flat unions to visit the base's base members (the previous
commit merely added them to the struct). Same test case.

Patch's effect on visit_type_UserDefFlatUnion():

static void visit_type_UserDefFlatUnion_fields(Visitor *m, UserDefFlatUnion **obj, Error **errp)
{
Error *err = NULL;

+ visit_type_int(m, &(*obj)->integer, "integer", &err);
+ if (err) {
+ goto out;
+ }
visit_type_str(m, &(*obj)->string, "string", &err);
if (err) {
goto out;

Test cases updated for the bug fix.

Fixes alternates to generate a visitor for their implicit enumeration
type. None of them are currently used, obviously. Example:
block-core.json's BlockdevRef now generates
visit_type_BlockdevRefKind().

Code is generated in a different order now, and therefore has got a
few new forward declarations. Doesn't matter.

The guard QAPI_VISIT_BUILTIN_VISITOR_DECL is renamed to
QAPI_VISIT_BUILTIN.

The previous commit's two ugly special cases exist here, too. Mark
both TODO.

Backports commit 441cbac0c7e641780decbc674a9a68c6a5200f71 from qemu
2018-02-19 17:04:59 -05:00
Markus Armbruster 03634275b1
qapi-types: Convert to QAPISchemaVisitor, fixing flat unions
Fixes flat unions to get the base's base members. Test case is from
commit 2fc0043, in qapi-schema-test.json:

{ 'union': 'UserDefFlatUnion',
'base': 'UserDefUnionBase',
'discriminator': 'enum1',
'data': { 'value1' : 'UserDefA',
'value2' : 'UserDefB',
'value3' : 'UserDefB' } }

{ 'struct': 'UserDefUnionBase',
'base': 'UserDefZero',
'data': { 'string': 'str', 'enum1': 'EnumOne' } }

{ 'struct': 'UserDefZero',
'data': { 'integer': 'int' } }

Patch's effect on UserDefFlatUnion:

struct UserDefFlatUnion {
/* Members inherited from UserDefUnionBase: */
+ int64_t integer;
char *string;
EnumOne enum1;
/* Own members: */
union { /* union tag is @enum1 */
void *data;
UserDefA *value1;
UserDefB *value2;
UserDefB *value3;
};
};

Flat union visitors remain broken. They'll be fixed next.

Code is generated in a different order now, but that doesn't matter.

The two guards QAPI_TYPES_BUILTIN_STRUCT_DECL and
QAPI_TYPES_BUILTIN_CLEANUP_DECL are replaced by just
QAPI_TYPES_BUILTIN.

Two ugly special cases for simple unions now stand out like sore
thumbs:

1. The type tag is named 'type' everywhere, except in generated C,
where it's 'kind'.

2. QAPISchema lowers simple unions to semantically equivalent flat
unions. However, the C generated for a simple unions differs from
the C generated for its equivalent flat union, and we therefore
need special code to preserve that pointless difference for now.

Mark both TODO.

Backports commit 2b162ccbe875e5323fc04c1009addbdea4d35220 from qemu
2018-02-19 16:58:34 -05:00
Lioncash 72f6d68002
qapi: New QAPISchemaVisitor
The visitor will help keeping the code generation code simple and
reasonably separated from QAPISchema details.

Backports commit 3f7dc21bee1e930d5cccf607b8f83831c3bbdb09 from qemu
2018-02-19 16:50:02 -05:00
Markus Armbruster 8ce395e073
qapi: QAPISchema code generation helper methods
New methods c_name(), c_type(), c_null(), json_type(),
alternate_qtype().

Backports commit f51d8c3db11b0f3052b3bb4b8b0c7f0bc76f7136 from qemu
2018-02-19 16:47:35 -05:00
Markus Armbruster a3660e451d
qapi: New QAPISchema intermediate reperesentation
The QAPI code generators work with a syntax tree (nested dictionaries)
plus a few symbol tables (also dictionaries) on the side.

They have clearly outgrown these simple data structures. There's lots
of rummaging around in dictionaries, and information is recomputed on
the fly. For the work I'm going to do, I want more clearly defined
and more convenient interfaces.

Going forward, I also want less coupling between the back-ends and the
syntax tree, to make messing with the syntax easier.

Create a bunch of classes to represent QAPI schemata.

Have the QAPISchema initializer call the parser, then walk the syntax
tree to create the new internal representation, and finally perform
semantic analysis.

Shortcut: the semantic analysis still relies on existing check_exprs()
to do the actual semantic checking. All this code needs to move into
the classes. Mark as TODO.

Simple unions are lowered to flat unions. Flat unions and structs are
represented as a more general object type.

Catching name collisions in generated code would be nice. Mark as
TODO.

We generate array types eagerly, even though most of them aren't used.
Mark as TODO.

Nothing uses the new intermediate representation just yet, thus no
change to generated files.

Backports commit ac88219a6c78302c693fb60fe6cf04358540fbce from qemu
2018-02-19 16:43:30 -05:00
Markus Armbruster 448a2e95ac
qapi: Rename class QAPISchema to QAPISchemaParser
I want to name a new class QAPISchema.

While there, make it a new-style class.

Backports commit a4bcb2080d5c1d08bab512d76fb260296e2cae74 from qemu
2018-02-19 16:41:00 -05:00
Daniel P. Berrange fe90a5c914
qapi: allow override of default enum prefix naming
The camel_to_upper() method applies some heuristics to turn
a mixed case type name into an all-uppercase name. This is
used for example, to generate enum constant name prefixes.

The heuristics don't also generate a satisfactory name
though. eg

{ 'enum': 'QCryptoTLSCredsEndpoint',
'data': ['client', 'server']}

Results in Q_CRYPTOTLS_CREDS_ENDPOINT_CLIENT. This has
an undesirable _ after the initial Q and is missing an
_ between the CRYPTO & TLS strings.

Rather than try to add more and more heuristics to try
to cope with this, simply allow the QAPI schema to
specify the desired enum constant prefix explicitly.

eg

{ 'enum': 'QCryptoTLSCredsEndpoint',
'prefix': 'QCRYPTO_TLS_CREDS_ENDPOINT',
'data': ['client', 'server']}

Now gives the QCRYPTO_TLS_CREDS_ENDPOINT_CLIENT name.

Backports commit 351d36e454cddc67a1675740916636a7ccbf1c4b from qemu
2018-02-19 16:38:48 -05:00
Markus Armbruster d59948f245
qapi: Fix cgen() for Python older than 2.7
A feature new in Python 2.7 crept into commit 77e703b: re.subn()'s
fifth argument.  Avoid that, use re.compile().

Backports commit 2752e5bedb26fa0c7291f810f9f534b688b2f1d2 from qemu
2018-02-19 16:34:17 -05:00
Markus Armbruster 809b439562
qapi: Generators crash when --output-dir isn't given, fix
Backports commit c4f498fe8532cdacc609262b104322911108df54 from qemu
2018-02-19 16:32:50 -05:00
Markus Armbruster 741783073f
qapi: Simplify error reporting for array types
check_type() first checks and peels off the array type, then checks
the element type. For two out of four error messages, it takes pains
to report errors for "array of T" instead of just T. Odd. Let's
examine the errors.

* Unknown element type, e.g.
tests/qapi-schema/args-array-unknown.json:

Member 'array' of 'data' for command 'oops' uses unknown type
'array of NoSuchType'

To make sense of this, you need to know that 'array of NoSuchType'
refers to '[NoSuchType]'. Easy enough. However, simply reporting

Member 'array' of 'data' for command 'oops' uses unknown type
'NoSuchType'

is at least as easy to understand.

* Element type's meta-type is inadmissible, e.g.
tests/qapi-schema/returns-whitelist.json:

'returns' for command 'no-way-this-will-get-whitelisted' cannot
use built-in type 'array of int'

'array of int' is technically not a built-in type, but that's
pedantry. However, simply reporting

'returns' for command 'no-way-this-will-get-whitelisted' cannot
use built-in type 'int'

avoids the issue, and is at least as easy to understand.

* The remaining two errors are unreachable, because the array checking
ensures that value is a string.

Thus, reporting some errors for "array of T" instead of just T works,
but doesn't really improve things. Drop it.

Backports commit eddf817bd823a90df209dfbdc2a0b2ec33b7cb77 from qemu
2018-02-19 16:31:44 -05:00
Markus Armbruster 630b56b4a2
qapi: Fix errors for non-string, non-dictionary members
Backports commit c6b71e5ae73802057d700e2419b80aef1651f213 from qemu
2018-02-19 16:30:50 -05:00
Markus Armbruster 16619a738e
qapi: Drop one of two "simple union must not have base" checks
The first check ensures the second one can't trigger. Drop the first
one, because the second one is in a more logical place, and emits a
nicer error message.

Backports commit 65fbe125451da9421070ab03944c9600a264eefc from qemu
2018-02-19 16:29:58 -05:00
Markus Armbruster 46cf5ed6ae
qapi: Generated code cleanup
Clean up white-space, brace placement, and superfluous #ifdef
QAPI_TYPES_BUILTIN_CLEANUP_DEF.

Backports commit 3a864e7c52af15017d5082a9ee39a7919f46d2b5 from qemu
2018-02-19 16:29:13 -05:00
Markus Armbruster c5e8b70b5a
qapi: Command returning anonymous type doesn't work, outlaw
Reproducer: with

{ 'command': 'user_def_cmd4', 'returns': { 'a': 'int' } }

added to qapi-schema-test.json, qapi-commands.py dies when it tries to
generate the command handler function

Traceback (most recent call last):
File "/work/armbru/qemu/scripts/qapi-commands.py", line 359, in <module>
ret = generate_command_decl(cmd['command'], arglist, ret_type) + "\n"
File "/work/armbru/qemu/scripts/qapi-commands.py", line 29, in generate_command_decl
ret_type=c_type(ret_type), name=c_name(name),
File "/work/armbru/qemu/scripts/qapi.py", line 927, in c_type
assert isinstance(value, str) and value != ""
AssertionError

because the return type doesn't exist.

Simply outlaw this usage, and drop or dumb down test cases accordingly.

Backports commit 9b090d42aea9a0abbf39a1d75561a186057b5fe6 from qemu
2018-02-19 16:24:14 -05:00
Markus Armbruster 8d992dfa48
qapi: Fix to reject union command and event arguments
A command's or event's 'data' must be a struct type, given either as a
dictionary, or as struct type name.

Commit dd883c6 tightened the checking there, but not enough: we still
accept 'union'. Fix to reject it.

We may want to support union types there, but we'll have to extend
qapi-commands.py and qapi-events.py for it.

Backports commit 315932b5edb86597adafbd1faa2d29c46499d8c3 from qemu
2018-02-19 16:23:24 -05:00
Markus Armbruster eed32a1c57
qapi: Document flaws in checking of names
We don't actually enforce our "other than downstream extensions [...],
all names should begin with a letter" rule. Add a FIXME.

We should reject names that differ only in '_' vs. '.' vs. '-',
because they're liable to clash in generated C. Add a FIXME.

Backports commit d90675fa4bc256238b3dd3a7fdd5f9029eca00b8 from qemu
2018-02-19 16:22:35 -05:00
Eric Blake 6e85e420fb
qapi: Document shortcoming with union 'data' branch
Add a FIXME to remind us to fully audit whether removing the
'void *data' branch of each qapi union type can be done safely.

Backports commit ca56a822dd538017715345cbbe1f8829e0cc2742 from qemu
2018-02-19 16:21:22 -05:00
Markus Armbruster 94b19608af
qapi-visit: Fix two name arguments passed to visitors
The generated code passes mangled schema names to visit_type_enum()
and union's visit_start_struct(). Fix it to pass the names
unadulterated, like we do everywhere else.

Only qapi-schema-test.json actually has names where this makes a
difference: enum __org.qemu_x-Enum, flat union __org.qemu_x-Union2,
simple union __org.qemu_x-Union1 and its implicit enum
__org.qemu_x-Union1Kind.

Backports commit 40b3adec13a9e022ff5a2e2b81c243fc0a026746 from qemu
2018-02-19 16:20:10 -05:00
Markus Armbruster 54ce4b3f00
qapi-visit: Replace list implicit_structs by set
Use set because that's what it is. While there, rename to
implicit_structs_seen.

Backports commit 8c07eddc619d618965fdd7a96bfe3b5c59f42b52 from qemu
2018-02-19 16:17:37 -05:00
Markus Armbruster 23d14a2921
qapi-visit: Fix generated code when schema has forward refs
The visit_type_implicit_FOO() are generated on demand, right before
their first use. Used by visit_type_STRUCT_fields() when STRUCT has
base FOO, and by visit_type_UNION() when flat UNION has member a FOO.

If the schema defines FOO after its first use as struct base or flat
union member, visit_type_implicit_FOO() calls
visit_type_implicit_FOO() before its definition, which doesn't
compile.

Rearrange qapi-schema-test.json to demonstrate the bug.

Fix by generating the necessary forward declaration.

Backports commit 8c3f8e77215bfedb7854221868f655e148506936 from qemu
2018-02-19 16:16:19 -05:00
Markus Armbruster 389afaa743
qapi: Generate a nicer struct for flat unions
The struct generated for a flat union is weird: the members of its
base are at the end, except for the union tag, which is at the
beginning.

Example: qapi-schema-test.json has

{ 'struct': 'UserDefUnionBase',
'data': { 'string': 'str', 'enum1': 'EnumOne' } }

{ 'union': 'UserDefFlatUnion',
'base': 'UserDefUnionBase',
'discriminator': 'enum1',
'data': { 'value1' : 'UserDefA',
'value2' : 'UserDefB',
'value3' : 'UserDefB' } }

We generate:

struct UserDefFlatUnion
{
EnumOne enum1;
union {
void *data;
UserDefA *value1;
UserDefB *value2;
UserDefB *value3;
};
char *string;
};

Change to put all base members at the beginning, unadulterated. Not
only is this easier to understand, it also permits casting the flat
union to its base, if that should become useful.

We now generate:

struct UserDefFlatUnion
{
/* Members inherited from UserDefUnionBase: */
char *string;
EnumOne enum1;
/* Own members: */
union { /* union tag is @enum1 */
void *data;
UserDefA *value1;
UserDefB *value2;
UserDefB *value3;
};
};

Backports commit 1e6c1616a91cdcbe9a8387541f7689b8c11632aa from qemu
2018-02-19 16:14:51 -05:00
Markus Armbruster 8b7252d8c8
qapi: Fix generated code when flat union has member 'kind'
A flat union's tag member gets renamed to 'kind' in the generated
code. Breaks when another member named 'kind' exists.

Example, adapted from qapi-schema-test.json:

{ 'struct': 'UserDefUnionBase',
'data': { 'kind': 'str', 'enum1': 'EnumOne' } }

We generate:

struct UserDefFlatUnion
{
EnumOne kind;
union {
void *data;
UserDefA *value1;
UserDefB *value2;
UserDefB *value3;
};
char *kind;
};

Kill the silly rename.

Backports commit 0f61af3eb396ae163cd1572ce12e05f5d08d7c15 from qemu
2018-02-19 16:13:07 -05:00
Markus Armbruster b190e4887e
qapi: Drop unused and useless parameters and variables
gen_sync_call()'s parameter indent is useless: gen_sync_call() uses it
only as optional argument for push_indent() and pop_indent(), their
default is four, and gen_sync_call()'s only caller passes four. Drop
the parameter.

gen_visitor_input_containers_decl()'s parameter obj is always
"QOBJECT(args)". Use that, and drop the parameter.

Drop unused parameters of gen_marshal_output(),
gen_marshal_input_decl(), generate_visit_struct_body(),
generate_visit_list(), generate_visit_enum(), generate_declaration(),
generate_enum_declaration(), generate_decl_enum().

Drop unused variables in generate_event_enum_lookup(),
generate_enum_lookup(), generate_visit_struct_fields(), check_event().

Backports commit 5aa05d3f72e556752167f7005d6a3dea0f4432c5 from qemu
2018-02-19 16:11:35 -05:00