For contiguous predicated memory operations, we want to
minimize the number of tlb lookups performed. We have
open-coded this for sve_ld1_r, but for correctness with
MTE we will need this for all of the memory operations.
Create a structure that holds the bounds of active elements,
and metadata for two pages. Add routines to find those
active elements, lookup the pages, and run watchpoints
for those pages.
Temporarily mark the functions unused to avoid Werror.
Backports commit b4cd95d2f4c7197b844f51b29871d888063ea3e7 from qemu
Use the "normal" memory access functions, rather than the
softmmu internal helper functions directly.
Since fb901c9, cpu_mem_index is now a simple extract
from env->hflags and not a large computation. Which means
that it's now more work to pass around this value than it
is to recompute it.
This only adjusts the primitives, and does not clean up
all of the uses within sve_helper.c.
Now that we know that the operation is on a single page,
we need not loop over pages while probing.
Backports commit e26d0d226892f67435cadcce86df0ddfb9943174 from qemu
No host backend support yet, but the interfaces for rotlv
and rotrv are in place.
Reviewed-by: Alex Bennée <alex.bennee@linaro.org>
Signed-off-by: Richard Henderson <richard.henderson@linaro.org>
---
v3: Drop the generic expansion from rot to shift; we can do better
for each backend, and then this code becomes unused.
Backports commit 5d0ceda902915e3f0e21c39d142c92c4e97c3ebb from qemu
No host backend support yet, but the interfaces for rotli
are in place. Canonicalize immediate rotate to the left,
based on a survey of architectures, but provide both left
and right shift interfaces to the translators.
Backports commit b0f7e7444c03da17e41bf327c8aea590104a28ab from qemu
The existing f{32,64}_addsub_post test, which checks for zero
inputs, is identical to f{32,64}_mul_fast_test. Which means
we can eliminate the fast_test/fast_op hooks in favor of
reusing the same post hook.
This means we have one fewer test along the fast path for multiply.
Backports commit b240c9c497b9880ac0ba29465907d5ebecd48083 from qemu
The commentary talks about "in concert with the addresses
assigned in the relevant linker script", except there is no
linker script for softmmu, nor has there been for some time.
(Do not confuse the user-only linker script editing that was
removed in the previous patch, because user-only does not
use this code_gen_buffer allocation method.)
Backports commit 64547a3bb6c92781372994e4e9b25a89f6c88074 from qemu
Partial cleanup from the CONFIG_VECTOR16 removal.
Replace DO_CMP0 with its scalar expansion, a simple negation.
Backports commit 0270bd503e3699b7202200a2d693ad1feb57473f from qemu
Partial cleanup from the CONFIG_VECTOR16 removal.
Replace the DUP* expansions with the scalar argument.
Backports commit 0a83e43a9ee624b44da61514db9b77d86e74e8c2 from qemu
Partial cleanup from the CONFIG_VECTOR16 removal.
Replace the vec* types with their scalar expansions.
Backports commit 6c7ab3015ac498181444deff55dcc8fd43ad468c from qemu
The comment in tcg-runtime-gvec.c about CONFIG_VECTOR16 says that
tcg-op-gvec.c has eliminated size 8 vectors, and only passes on
multiples of 16. This may have been true of the first few operations,
but is not true of all operations.
In particular, multiply, shift by scalar, and compare of 8- and 16-bit
elements are not expanded inline if host vector operations are not
supported.
For an x86_64 host that does not support AVX, this means that we will
fall back to the helper, which will attempt to use SSE instructions,
which will SEGV on an invalid 8-byte aligned memory operation.
This patch simply removes the CONFIG_VECTOR16 code and configuration
without further simplification.
Buglink: https://bugs.launchpad.net/bugs/1863508
Backports commit 43d1ccd2a02fadf3c36b46a8c31125a28864d141 from qemu
Under some circumstances, the PC is not fixed up properly when
returning from the execution of a block in cpu_tb_exec. This appears
to be caused by the resetting of the PC from the tb.
This change removes the additional fixup in the cases where there
is code tracing or timing active. Either of these cases would result
in the wrong PC being reported.
Closes unicorn-engine#1105.
Backports commit b59632fb645d456338472e3d757c065c0ed74ad5 from unicorn
Calling emu_stop and causing the pc value to be incorrect after the end of the run. (#1157)
Backports commit 83887b8193dfeca3e5e8da851b41f874bcd0514e from unicorn.
* fix a mem-leak.
* check the uc and l1_map before using them.
* fix multi-level free bug.
* Add pointer check.
Backports commit 79d89e5d3b83c6ee5d523738bc488d1e44b06f6a from unicorn.
* first draft for an invalid instruction hook
* Fixed documentation on return value of invalid insn hook
Backports commit 07f94ad1fc62293cac330df9714d739be6354926 from unicorn
Background: s390x implements Low-Address Protection (LAP). If LAP is
enabled, writing to effective addresses (before any translation)
0-511 and 4096-4607 triggers a protection exception.
So we have subpage protection on the first two pages of every address
space (where the lowcore - the CPU private data resides).
By immediately invalidating the write entry but allowing the caller to
continue, we force every write access onto these first two pages into
the slow path. we will get a tlb fault with the specific accessed
addresses and can then evaluate if protection applies or not.
We have to make sure to ignore the invalid bit if tlb_fill() succeeds.
Backports commit f52bfb12143e29d7c8bd827bdb751aee47a9694e from qemu
... similar to tlb_vaddr_to_host(); however, allow access to the host
page except when TLB_NOTDIRTY or TLB_MMIO is set.
Backports commit fef39ccd567032d3ad520ed80f3576068e6eb2e3 from qemu
Let size > 0 indicate a promise to write to those bytes.
Check for write watchpoints in the probed range.
Backports commit 03a981893c99faba84bb373976796ad7dce0aecc from qemu
The raising of exceptions from check_watchpoint, buried inside
of the I/O subsystem, is fundamentally broken. We do not have
the helper return address with which we can unwind guest state.
Replace PHYS_SECTION_WATCH and io_mem_watch with TLB_WATCHPOINT.
Move the call to cpu_check_watchpoint into the cputlb helpers
where we do have the helper return address.
This allows watchpoints on RAM to bypass the full i/o access path.
Backports commit 50b107c5d617eaf93301cef20221312e7a986701 from qemu
We had two different mechanisms to force a recheck of the tlb.
Before TLB_RECHECK was introduced, we had a PAGE_WRITE_INV bit
that would immediate set TLB_INVALID_MASK, which automatically
means that a second check of the tlb entry fails.
We can use the same mechanism to handle small pages.
Conserve TLB_* bits by removing TLB_RECHECK.
Backports commit 30d7e098d5c38644359820317fcf72e3e129ec53 from qemu
We have already aligned page2 to the start of the next page.
There is no reason to do that a second time.
Backports commit 5787585d0406cfd54dda0c71ea1a603347ce6e71 from qemu
We are currently passing the size of the full write to
the tlb_fill for the second page. Instead pass the real
size of the write to that page.
This argument is unused within all tlb_fill, except to be
logged via tracing, so in practice this makes no difference.
But in a moment we'll need the value of size2 for watchpoints,
and if we've computed the value we might as well use it.
Backports commit 8f7cd2ad4acd01242d00807e231097b3de9f0930 from qemu
Notice new attribute, byte swap, and force the transaction through the
memory slow path.
Required by architectures that can invert endianness of memory
transaction, e.g. SPARC64 has the Invert Endian TTE bit.
Backports commit a26fc6f5152b47f1d7ed928f9c9d462d01ff1624 from qemu
Now that MemOp has been pushed down into the memory API, and
callers are encoding endianness, we can collapse byte swaps
along the I/O path into the accelerator and target independent
adjust_endianness.
Collapsing byte swaps along the I/O path enables additional endian
inversion logic, e.g. SPARC64 Invert Endian TTE bit, with redundant
byte swaps cancelling out.
Backports commit 9bf825bf3df4ebae3af51566c8088e3f1249a910 from qemu
Preparation for collapsing the two byte swaps adjust_endianness and
handle_bswap into the former.
Backports commit be5c4787e9a6eed12fd765d9e890f7cc6cd63220 from qemu
Preparation for collapsing the two byte swaps adjust_endianness and
handle_bswap into the former.
Call memory_region_dispatch_{read|write} with endianness encoded into
the "MemOp op" operand.
This patch does not change any behaviour as
memory_region_dispatch_{read|write} is yet to handle the endianness.
Once it does handle endianness, callers with byte swaps can collapse
them into adjust_endianness.
Backports commit d5d680cacc66ef7e3c02c81dc8f3a34eabce6dfe from qemu
The memory_region_dispatch_{read|write} operand "unsigned size" is
being converted into a "MemOp op".
Convert interfaces by using no-op size_memop.
After all interfaces are converted, size_memop will be implemented
and the memory_region_dispatch_{read|write} operand "unsigned size"
will be converted into a "MemOp op".
As size_memop is a no-op, this patch does not change any behaviour.
Backports commit 4cbb198eefef41bbca703605c78875fd4fec6ef6 from qemu
Preparation for collapsing the two byte swaps, adjust_endianness and
handle_bswap, along the I/O path.
Target dependant attributes are conditionalized upon NEED_CPU_H.
Backports commit 14776ab5a12972ea439c7fb2203a4c15a09094b4 from qemu
The loop is written with scalars, not vectors.
Use the correct type when incrementing.
Fixes: 5ee5c14cacd
Backports commit 899f08ad1d1231dbbfa67298413f05ed2679fb02 from qemu
While size_t is defined to happily access the biggest host object this
isn't the case when generating masks for 64 bit guests on 32 bit
hosts. Otherwise we end up truncating the address when we fall back to
our unaligned helper.
Fixes: https://bugs.launchpad.net/qemu/+bug/1831545
Backports commit ab7a2009df66241a3742cbdfe8f9a1f66c6af21f from qemu
When running on 32 bit TCG backends a wide unaligned load ends up
truncating data before returning to the guest. We specifically have
the return type as uint64_t to avoid any premature truncation so we
should use the same for the interim types.
Fixes: https://bugs.launchpad.net/qemu/+bug/1830872
Fixes: eed5664238e
Backports commit 8c79b288513587e960b6b7257a9d955d5592f209 from qemu
Amusingly, we had already ignored the comment to keep this value
at the end of CPUState. This restores the minimum negative offset
from TCG_AREG0 for code generation.
For the couple of uses within qom/cpu.c, without NEED_CPU_H, add
a pointer from the CPUState object to the IcountDecr object within
CPUNegativeOffsetState.
Backports commit 5e1401969b25f676fee6b1c564441759cf967a43 from qemu
Now that we have both ArchCPU and CPUArchState, we can define
this generically instead of via macro in each target's cpu.h.
Backports commit 29a0af618ddd21f55df5753c3e16b0625f534b3c from qemu
Especially for guests with large numbers of tlbs, like ARM or PPC,
we may well not use all of them in between flush operations.
Remember which tlbs have been used since the last flush, and
avoid any useless flushing.
Backports much of 3d1523ced6060cdfe9e768a814d064067ccabfe5 from qemu
along with a bunch of updating changes.
This operation performs d = (b & a) | (c & ~a), and is present
on a majority of host vector units. Include gvec expanders.
Backports commit 38dc12947ec9106237f9cdbd428792c985cd86ae from qemu
Most of the existing users would continue around a loop which
would fault the tlb entry in via a normal load/store.
But for AArch64 SVE we have an existing emulation bug wherein we
would mark the first element of a no-fault vector load as faulted
(within the FFR, not via exception) just because we did not have
its address in the TLB. Now we can properly only mark it as faulted
if there really is no valid, readable translation, while still not
raising an exception. (Note that beyond the first element of the
vector, the hardware may report a fault for any reason whatsoever;
with at least one element loaded, forward progress is guaranteed.)
Backports commit 4811e9095c0491bc6f5450e5012c9c4796b9e59d from qemu
We can now use the CPUClass hook instead of a named function.
Create a static tlb_fill function to avoid other changes within
cputlb.c. This also isolates the asserts within. Remove the
named tlb_fill function from all of the targets.
Backports commit c319dc13579a92937bffe02ad2c9f1a550e73973 from qemu
The gvec expanders perform a modulo on the shift count. If the target
requires alternate behaviour, then it cannot use the generic gvec
expanders anyway, and will have to have its own custom code.
Backports commit 5ee5c14cacda27e904cd6b0d9e7ffe1acff42838 from qemu
This is less tricky than for loads, because we always fall
back to single byte stores to implement unaligned stores.
Backports commit 4601f8d10d7628bcaf2a8179af36e04b42879e91 from qemu
If we attempt to recurse from load_helper back to load_helper,
even via intermediary, we do not get all of the constants
expanded away as desired.
But if we recurse back to the original helper (or a shim that
has a consistent function signature), the operands are folded
away as desired.
Backports commit 2dd926067867c2dd19e66d31a7990e8eea7258f6 from qemu
Going to approach this problem via __attribute__((always_inline))
instead, but full conversion will take several steps.
Backports commit fc1bc777910dc14a3db4e2ad66f3e536effc297d from qemu
Having this in io_readx/io_writex meant that we forgot to
re-compute index after tlb_fill. It also means we can use
the normal aligned memory load path. It also fixes a bug
in that we had cached a use of index across a tlb_fill.
Backports commit f1be36969de2fb9b6b64397db1098f115210fcd9 from qemu
Instead of expanding a series of macros to generate the load/store
helpers we move stuff into common functions and rely on the compiler
to eliminate the dead code for each variant.
Backports commit eed5664238ea5317689cf32426d9318686b2b75c from qemu
Now that we have curr_cflags, we can include CF_USE_ICOUNT
early and then remove it as necessary.
Backports commit 416986d3f97329655e30da7271a2d11c6d707b06 from qemu
