Commit graph

2822 commits

Author SHA1 Message Date
Markus Armbruster 23d14a2921
qapi-visit: Fix generated code when schema has forward refs
The visit_type_implicit_FOO() are generated on demand, right before
their first use. Used by visit_type_STRUCT_fields() when STRUCT has
base FOO, and by visit_type_UNION() when flat UNION has member a FOO.

If the schema defines FOO after its first use as struct base or flat
union member, visit_type_implicit_FOO() calls
visit_type_implicit_FOO() before its definition, which doesn't
compile.

Rearrange qapi-schema-test.json to demonstrate the bug.

Fix by generating the necessary forward declaration.

Backports commit 8c3f8e77215bfedb7854221868f655e148506936 from qemu
2018-02-19 16:16:19 -05:00
Markus Armbruster 389afaa743
qapi: Generate a nicer struct for flat unions
The struct generated for a flat union is weird: the members of its
base are at the end, except for the union tag, which is at the
beginning.

Example: qapi-schema-test.json has

{ 'struct': 'UserDefUnionBase',
'data': { 'string': 'str', 'enum1': 'EnumOne' } }

{ 'union': 'UserDefFlatUnion',
'base': 'UserDefUnionBase',
'discriminator': 'enum1',
'data': { 'value1' : 'UserDefA',
'value2' : 'UserDefB',
'value3' : 'UserDefB' } }

We generate:

struct UserDefFlatUnion
{
EnumOne enum1;
union {
void *data;
UserDefA *value1;
UserDefB *value2;
UserDefB *value3;
};
char *string;
};

Change to put all base members at the beginning, unadulterated. Not
only is this easier to understand, it also permits casting the flat
union to its base, if that should become useful.

We now generate:

struct UserDefFlatUnion
{
/* Members inherited from UserDefUnionBase: */
char *string;
EnumOne enum1;
/* Own members: */
union { /* union tag is @enum1 */
void *data;
UserDefA *value1;
UserDefB *value2;
UserDefB *value3;
};
};

Backports commit 1e6c1616a91cdcbe9a8387541f7689b8c11632aa from qemu
2018-02-19 16:14:51 -05:00
Markus Armbruster 8b7252d8c8
qapi: Fix generated code when flat union has member 'kind'
A flat union's tag member gets renamed to 'kind' in the generated
code. Breaks when another member named 'kind' exists.

Example, adapted from qapi-schema-test.json:

{ 'struct': 'UserDefUnionBase',
'data': { 'kind': 'str', 'enum1': 'EnumOne' } }

We generate:

struct UserDefFlatUnion
{
EnumOne kind;
union {
void *data;
UserDefA *value1;
UserDefB *value2;
UserDefB *value3;
};
char *kind;
};

Kill the silly rename.

Backports commit 0f61af3eb396ae163cd1572ce12e05f5d08d7c15 from qemu
2018-02-19 16:13:07 -05:00
Markus Armbruster b190e4887e
qapi: Drop unused and useless parameters and variables
gen_sync_call()'s parameter indent is useless: gen_sync_call() uses it
only as optional argument for push_indent() and pop_indent(), their
default is four, and gen_sync_call()'s only caller passes four. Drop
the parameter.

gen_visitor_input_containers_decl()'s parameter obj is always
"QOBJECT(args)". Use that, and drop the parameter.

Drop unused parameters of gen_marshal_output(),
gen_marshal_input_decl(), generate_visit_struct_body(),
generate_visit_list(), generate_visit_enum(), generate_declaration(),
generate_enum_declaration(), generate_decl_enum().

Drop unused variables in generate_event_enum_lookup(),
generate_enum_lookup(), generate_visit_struct_fields(), check_event().

Backports commit 5aa05d3f72e556752167f7005d6a3dea0f4432c5 from qemu
2018-02-19 16:11:35 -05:00
Markus Armbruster cf9d457200
qapi: Reject -p arguments that break qapi-event.py
qapi-event.py breaks when you ask for a funny prefix like '@'.
Protect it.

Backports commit 1cf47a15f18312436c7fa2d97be5fbe6df0292f5 from qemu
2018-02-19 16:07:40 -05:00
Markus Armbruster 2c3f6e4175
qapi-event: Clean up how name of enum QAPIEvent is made
Use c_name() instead of ad hoc code. Doesn't upcase the -p prefix,
which is an improvement in my book. Unbreaks prefix containing '.',
but other funny characters remain broken. To be fixed next.

Backports commit 016a335bd8ca624f43adbb08fa1698c29ec52a1a from qemu
2018-02-19 16:06:35 -05:00
Markus Armbruster 6dcb71a788
qapi: Simplify guardname()
The guards around built-in declarations lose their _H. It never made
much sense anyway.

Backports commit 00dfc3b2c272d98556ec6095d56bdd8b036babf9 from qemu
2018-02-19 16:05:52 -05:00
Markus Armbruster e100831af9
qapi: Clean up cgen() and mcgen()
Commit 05dfb26 added eatspace stripping to mcgen(). Move it to
cgen(), just in case somebody gets tempted to use cgen() directly
instead of via mcgen().

cgen() indents blank lines. No such lines get generated right now,
but fix it anyway.

We use triple-quoted strings for program text, like this:

'''
Program text
any number of lines
'''

Keeps the program text relatively readable, but puts an extra newline
at either end. mcgen() "fixes" that by dropping the first and last
line outright. Drop only the newlines.

This unmasks a bug in qapi-commands.py: four quotes instead of three.
Fix it up.

Output doesn't change

Backports commit 77e703b861d34bb2879f3e845482d5cf0a3a0ad1 from qemu
2018-02-19 16:04:37 -05:00
Daniel P. Berrange 767e900547
qom: Make enum string tables const-correct
The enum string table parameters in various QOM/QAPI methods
are declared 'const char *strings[]'. This results in const
warnings if passed a variable that was declared as

   static const char * const strings[] = { .... };

Add the extra const annotation to the parameters, since
neither the string elements, nor the array itself should
ever be modified.

Backports commit 2e4450ff432daef524cb3557fca68a3b7b5c7823 from qemu
2018-02-19 16:02:23 -05:00
Markus Armbruster 21b2c489af
qapi-types: Bury code dead since commit 6b5abc7
Backports commit e1d4210c3a50059a3889cedc44a8aa193fa63d7d from qemu
2018-02-19 15:58:09 -05:00
Peter Maydell 9a3972b6b2
scripts/qapi-types.py: Add dummy member to empty structs
Make sure that all generated C structs have at least one field; this
avoids potential issues with attempting to malloc space for
zero-length structs in C (g_malloc(sizeof struct) would return NULL).
It also avoids an incompatibility with C++ (where an empty struct is
size 1); that isn't important to us now but might be in future.

Generated empty structures look like this:
    struct Abort
    {
        char qapi_dummy_field_for_empty_struct;
    };

This silences clang warnings like:
./qapi-types.h:3752:1: warning: empty struct has size 0 in C, size 1 in C++ [-Wextern-c-compat]
struct Abort
^

Backports commit 83ecb22ba2c91a4674ae109595a8ed1da8de4d7a from qemu
2018-02-19 15:56:31 -05:00
Markus Armbruster 5efb546d6f
qapi-types: Split generate_fwd_builtin() off generate_fwd_struct()
Backports commit c5ecd7e18f912ab5e91f09b0333fb07567885d42 from qemu
2018-02-19 15:51:44 -05:00
Markus Armbruster e78c14f6e4
qapi-types: Drop unused members parameters
Backports commit ae0a7a109037160465f55f8bab06897f0a904def from qemu
2018-02-19 15:50:19 -05:00
Markus Armbruster 9ce13ce6a3
qapi-types: Don't filter out expressions with 'gen'
Useless, because it can only occur in commands, and we're not dealing
with commands here.

Backports commit 4f3568002393380558705397bda4cd5f224ffe29 from qemu
2018-02-19 15:48:52 -05:00
Markus Armbruster ed3da56d26
qapi: Catch and reject flat union branch of array type
Backports commit f9a1427361fe06ac67480d580412dc4ed6f5d03b from qemu
2018-02-19 15:48:12 -05:00
Markus Armbruster 52e7d76d23
qapi: Better separate the different kinds of helpers
Insert comments to separate sections dealing with parsing, semantic
analysis, code generation, and so forth.

Move helpers to their proper section.

Backports commit 00e4b285a31d19dcd88bd46729c9e09bfc9cc7fd from qemu
2018-02-19 15:47:21 -05:00
Markus Armbruster 0a8ab4fc40
qapi: Move exprs checking from parse_schema() to check_exprs()
To have expression semantic analysis in one place rather than two.

Backports commit 4d076d67c2c74662db092ecf4f99600b18209b2e from qemu
2018-02-19 15:45:24 -05:00
Markus Armbruster 72e6966bba
qapi: Fix to reject stray 't', 'f' and 'n'
Screwed up in commit e53188a.

Backports commit e565d934d21e3544b820cd03b88061e71ab644a0 from qemu
2018-02-19 15:41:38 -05:00
Markus Armbruster de88f40dfd
qapi: Simplify inclusion cycle detection
We maintain a stack of filenames in include_hist for convenient cycle
detection.

As error_path() demonstrates, the same information is readily
available in the expr_info, so just use that, and drop include_hist.

Backports commit a1366087270b312d94ff8c4031395a4218f160d4 from qemu
2018-02-19 15:40:44 -05:00
Markus Armbruster b387a62f73
qapi: Fix file name in error messages for included files
We print the name as it appears in the include expression. Tools
processing error messages want it relative to the working directory.
Make it so.

Backports commit 8608d2525186062099a38971c276752e7a38903a from qemu
2018-02-19 15:39:08 -05:00
Markus Armbruster a48f709cd8
qapi: Eliminate superfluous QAPISchema attribute input_dir
qapi: Improve a couple of confusing variable names

Backports commits 12c707944927b8aa42752198dcf419a0bafe5d33 and
54414047eca5bee7d5ba6e7af5fb251f8635896c from qemu
2018-02-19 15:37:52 -05:00
Markus Armbruster faffdb784a
qapi: Drop pointless flush() before close()
Backports commit 09896d3f48078a93e3d2dbd8ef86436b85ebda7c from qemu
2018-02-19 15:33:00 -05:00
Markus Armbruster 2c9ed3c379
qapi: Factor open_output(), close_output() out of generators
Backports commit 12f8e1b9ff57e99dafbb13f89cd5a99ad5c28527 from qemu
2018-02-19 15:32:28 -05:00
Markus Armbruster d77e0dd040
qapi: Turn generators' mandatory option -i into an argument
Mandatory option is silly, and the error handling is missing: the
programs crash when -i isn't supplied. Make it an argument, and check
it properly.

Backports commit 16d80f61814745bd3f5bb9f47ae3b00edf9e1e45 from qemu
2018-02-19 15:22:27 -05:00
Markus Armbruster dd67bbeb3b
qapi: Fix generators to report command line errors decently
Report to stderr, prefix with the program name. Also reject
extra arguments.

Backports commit b45409683e829770000a4560ed21e704f87df74c from qemu
2018-02-19 15:20:07 -05:00
Markus Armbruster 9415f6e863
qapi: Factor parse_command_line() out of the generators
Backports commit 2114f5a98d0d80774306279e1694de074ca86aa0 from qemu
2018-02-19 15:19:13 -05:00
Markus Armbruster f6a4f3033d
qapi: qapi-event.py option -b does nothing, drop it
Backports commit c70cef5bd48c7be603f75a7b5346db032a31b470 from qemu
2018-02-19 15:16:52 -05:00
Eric Blake 2dcd6722fa
qapi: Support downstream alternates
Enhance the testsuite to cover downstream alternates, including
whether the branch name or type is downstream. Update the
generator to mangle alternate names in the appropriate places.

Backports commit d1f07c86c05706facf950b0b0dba370f71fd5ef6 from qemu
2018-02-19 15:10:14 -05:00
Eric Blake 4292c61dbe
qapi: Support downstream flat unions
Enhance the testsuite to cover downstream flat unions, including
the base type, discriminator name and type, and branch name and
type. Update the generator to mangle the union names in the
appropriate places.

Backports commit 857af5f06c3fb097d1bb6bc8a23b9992aac99e75 from qemu
2018-02-19 15:06:36 -05:00
Eric Blake a2119bd210
qapi: Support downstream simple unions
Enhance the testsuite to cover downstream simple unions, including
when a union branch is a downstream name. Update the generator to
mangle the union names in the appropriate places.

Backports commit bb33729043ceda56b4068db13bdc17786ebd0ed0 from qemu
2018-02-19 15:05:10 -05:00
Eric Blake 2d14039f98
qapi: Support downstream structs
Enhance the testsuite to cover downstream structs, including struct
members and base structs. Update the generator to mangle the
struct names in the appropriate places.

Backports commit 83a02706bb1fd31c93eab755de543dfe228682d4 from qemu
2018-02-19 15:03:34 -05:00
Eric Blake 1a5b6a48d1
qapi: Support downstream enums
Enhance the testsuite to cover a downstream enum type and enum
string. Update the generator to mangle the enum name in the
appropriate places.

Backports commit fce384b8e5193e02421f6b2c2880f3684abcbdc0 from qemu
2018-02-19 15:01:08 -05:00
Eric Blake 2045cd0ada
qapi: Make c_type() consistently convert qapi names
Continuing the string of cleanups for supporting downstream names
containing '.', this patch focuses on ensuring c_type() can
handle a downstream name. This patch alone does not fix the
places where generator output should be calling this function
but was open-coding things instead, but it gets us a step closer.

In particular, the changes to c_list_type() and type_name() mean
that type_name(FOO) now handles the case when FOO contains '.',
'-', or is a ticklish identifier other than a builtin (builtins
are exempted because ['int'] must remain mapped to 'intList' and
not 'q_intList'). Meanwhile, ['unix'] now maps to 'q_unixList'
rather than 'unixList', to match the fact that 'unix' is ticklish;
however, our naming conventions state that complex types should
start with a capital, so no type name following conventions will
ever have the 'q_' prepended.

Likewise, changes to c_type() mean that c_type(FOO) properly
handles an enum or complex type FOO with '.' or '-' in the
name, or is a ticklish identifier (again, a ticklish identifier
as a type name violates conventions).

Backports commit c6405b54b7b09a876f2f2fba2aa6f8ac87189cb9 from qemu
2018-02-19 14:56:49 -05:00
Eric Blake 7ec0edc6c6
qapi: Tidy c_type() logic
c_type() is designed to be called on both string names and on
array designations, so 'name' is a bit misleading because it
operates on more than strings. Also, no caller ever passes
an empty string. Finally, + notation is a bit nicer to read
than '%s' % value for string concatenation.

Backports commit d557344628e32771f07e5b6a2a818ee3d8e7a65f from qemu
2018-02-19 14:53:56 -05:00
Markus Armbruster 5bc3d84705
qapi: Move camel_to_upper(), c_enum_const() to closely related code
Backports commit 849bc5382e42b3b9590c6a50ba30c2fd2450308c from qemu
2018-02-19 14:51:03 -05:00
Markus Armbruster e339979717
qapi: Simplify c_enum_const()
Backports commit 02e20c7e593363c564aae96e3c5bdc58630ce584 from qemu
2018-02-19 14:50:04 -05:00
Markus Armbruster b0adaeb172
qapi: Rename generate_enum_full_value() to c_enum_const()
Backports commit 7c81c61f9c2274f66ba947eafd9618d60da838a6 from qemu
2018-02-19 14:49:20 -05:00
Markus Armbruster 8c7bbc2dce
qapi: Rename _generate_enum_string() to camel_to_upper()
Backports commit fa6068a1e8ef3c878ac9ee2399bb01eeaf61c366 from qemu
2018-02-19 14:46:49 -05:00
Eric Blake f114c7b027
qapi: Rename identical c_fun()/c_var() into c_name()
Now that the two functions are identical, we only need one of them,
and we might as well give it a more descriptive name. Basically,
the function serves as the translation from a QAPI name into a
(portion of a) C identifier, without regards to whether it is a
variable or function name.

Backports commit 18df515ebbefa9f13474b128b8050d5fa346ea1e from qemu
2018-02-19 14:45:04 -05:00
Markus Armbruster baab1986a3
qapi: Fix C identifiers generated for names containing '.'
c_fun() maps '.' to '_', c_var() doesn't. Nothing prevents '.' in
QAPI names that get passed to c_var().

Which QAPI names get passed to c_fun(), to c_var(), or to both is not
obvious. Names of command parameters and struct type members get
passed to c_var().

c_var() strips a leading '*', but this cannot happen. c_fun()
doesn't.

Fix c_var() to work exactly like c_fun().

Perhaps they should be replaced by a single mapping function.

Backports commit 47299262de424af0cb69965d082e5e70b2314183 from qemu
2018-02-19 14:41:06 -05:00
Eric Blake 7a82e7ff73
qapi: Check for member name conflicts with a base class
Our type inheritance for both 'struct' and for flat 'union' merges
key/value pairs from the base class with those from the type in
question. Although the C code currently boxes things so that there
is a distinction between which member is referred to, the QMP wire
format does not allow passing a key more than once in a single
object. Besides, if we ever change the generated C code to not be
quite so boxy, we'd want to avoid duplicate member names there,
too.

Fix a testsuite entry added in an earlier patch, as well as adding
a couple more tests to ensure we have appropriate coverage. Ensure
that collisions are detected, regardless of whether there is a
difference in opinion on whether the member name is optional.

Backports commit ff55d72eaf9628e7d58e7b067b361cdbf789c9f4 from qemu
2018-02-19 14:38:46 -05:00
Eric Blake 90dfdc5278
qapi: Support (subset of) \u escapes in strings
The handling of \ inside QAPI strings was less than ideal, and
really only worked JSON's \/, \\, \", and our extension of \'
(an obvious extension, when you realize we use '' instead of ""
for strings). For other things, like '\n', it resulted in a
literal 'n' instead of a newline.

Of course, at the moment, we really have no use for escaped
characters, as QAPI has to map to C identifiers, and we currently
support ASCII only for that. But down the road, we may add
support for default values for string parameters to a command
or struct; if that happens, it would be nice to correctly support
all JSON escape sequences, such as \n or \uXXXX. This gets us
closer, by supporting Unicode escapes in the ASCII range.

Since JSON does not require \OCTAL or \xXX escapes, and our QMP
implementation does not understand them either, I intentionally
reject it here, but it would be an easy addition if we desired it.
Likewise, intentionally refusing the NUL byte means we don't have
to worry about C strings being shorter than the qapi input.

Backports commit a7f5966b297330f6492020019544ae87c45d699b from qemu
2018-02-19 14:36:54 -05:00
Eric Blake 937daf7d25
qapi: Drop dead visitor code related to nested structs
Now that we no longer have nested structs to visit, the use of
prefix strings is no longer required. Remove the code that is
no longer reachable.

Backports commit a82b982e2bddf7cd7cb490f83643e952e17d4523 from qemu
2018-02-19 14:35:55 -05:00
Eric Blake 8d2f349447
qapi: Drop support for inline nested types
A future patch will be using a 'name':{dictionary} entry in the
QAPI schema to specify a default value for an optional argument
(see previous commit messages for more details why); but existing
use of inline nested structs conflicts with that goal. Now that
all commands have been changed to avoid inline nested structs,
nuke support for them, and turn it into a hard error. Update the
testsuite to reflect tighter parsing rules.

Backports commit 6b5abc7df7ef9aadb3ff0eba6ccf4f1f0181e2e1 from qemu
2018-02-19 14:30:36 -05:00
Eric Blake 2d6d612c61
qapi: Forbid 'type' in schema
Referring to "type" as both a meta-type (built-in, enum, union,
alternate, or struct) and a specific type (the name that the
schema uses for declaring structs) is confusing. Finish up the
conversion to using "struct" in qapi schema by removing the hack
in the generator that allowed 'type'.

Backports commit 3e391d355644b2bff7c9f187759aadb46c6e051f from qemu
2018-02-19 14:23:21 -05:00
Eric Blake 341f4a437a
qapi: Use 'struct' instead of 'type' in schema
Referring to "type" as both a meta-type (built-in, enum, union,
alternate, or struct) and a specific type (the name that the
schema uses for declaring structs) is confusing. Do the bulk of
the conversion to "struct" in qapi schema, with a fairly
mechanical:

for f in `find -name '*.json'; do sed -i "s/'type'/'struct'/"; done

followed by manually filtering out the places where we have a
'type' embedded in 'data'. Then tweak a couple of tests whose
output changes slightly due to longer lines.

I also verified that the generated files for QMP and QGA (such
as qmp-commands.h) are the same before and after, as assurance
that I didn't leave in any accidental member name changes.

Backports commit 895a2a80e0e054f0d5d3715aa93d10d15e49f9f7 from qemu
2018-02-19 14:22:13 -05:00
Eric Blake b18ac34baa
qapi: Require ASCII in schema
Python 2 and Python 3 have a wild history of whether strings
default to ascii or unicode, where Python 3 requires checking
isinstance(foo, basestr) to cover all strings, but where that
code is not portable to Python 2. It's simpler to just state
that we don't care about Unicode strings, and to just always
use the simpler isinstance(foo, str) everywhere.

I'm no python expert, so I'm basing it on this conversation:
https://lists.gnu.org/archive/html/qemu-devel/2014-09/msg05278.html

Backports commit fe2a9303c9e511462f662a415c2e9d2defe9b7ca from qemu
2018-02-19 14:18:52 -05:00
Eric Blake 0fc76ffc1f
qapi: Prefer 'struct' over 'type' in generator
Referring to "type" as both a meta-type (built-in, enum, union,
alternate, or struct) and a specific type (the name that the
schema uses for declaring structs) is confusing. The confusion
is only made worse by the fact that the generator mostly already
refers to struct even when dealing with expr['type']. This
commit changes the generator to consistently refer to it as
struct everywhere, plus a single back-compat tweak that allows
accepting the existing .json files as-is, so that the meat of
this change is separate from the mindless churn of that change.

Fix the testsuite fallout for error messages that change, and
in some cases, become more legible. Improve comments to better
match our intentions where a struct (rather than any complex
type) is required. Note that in some cases, an error message
now refers to 'struct' while the schema still refers to 'type';
that will be cleaned up in the later commit to the schema.

Backports commit fd41dd4eae5f7ea92f10c04cb3f217727fcee91f from qemu
2018-02-19 14:13:32 -05:00
Eric Blake 06faf280f1
qapi: More rigorous checking for type safety bypass
Now that we have a way to validate every type, we can also be
stricter about enforcing that callers that want to bypass
type safety in generated code. Prior to this patch, it didn't
matter what value was associated with the key 'gen', but it
looked odd that 'gen':'yes' could result in bypassing the
generated code. These changes also enforce the changes made
earlier in the series for documentation and consolidation of
using '**' as the wildcard type, as well as 'gen':false as the
canonical spelling for requesting type bypass.

Note that 'gen':false is a one-way switch away from the default;
we do not support 'gen':true (similar for 'success-response').
In practice, this doesn't matter.

Backports commit 2cbf09925ad45401673a79ab77f67de2f04a826c from qemu
2018-02-19 14:09:03 -05:00
Eric Blake 0b9d15dd52
qapi: Whitelist commands that don't return dictionary
...or an array of dictionaries. Although we have to cater to
existing commands, returning a non-dictionary means the command
is not extensible (no new name/value pairs can be added if more
information must be returned in parallel). By making the
whitelist explicit, any new command that falls foul of this
practice will have to be self-documenting, which will encourage
developers to either justify the action or rework the design to
use a dictionary after all.

It's a little bit sloppy that we share a single whitelist among
three clients (it's too permissive for each). If this is a
problem, a future patch could tighten things by having the
generator take the whitelist as an argument (as in
scripts/qapi-commands.py --legacy-returns=...), or by having
the generator output C code that requires explicit use of the
whitelist (as in:
then having the callers define appropriate macros). But until
we need such fine-grained separation (if ever), this patch does
the job just fine.

Backports commit 10d4d997f86cf2a4ce89145df5658952d5722e56 from qemu
2018-02-19 14:06:23 -05:00