Commit graph

61 commits

Author SHA1 Message Date
Richard Henderson 94b0876f15 target/arm: Add sve infrastructure for page lookup
For contiguous predicated memory operations, we want to
minimize the number of tlb lookups performed. We have
open-coded this for sve_ld1_r, but for correctness with
MTE we will need this for all of the memory operations.

Create a structure that holds the bounds of active elements,
and metadata for two pages. Add routines to find those
active elements, lookup the pages, and run watchpoints
for those pages.

Temporarily mark the functions unused to avoid Werror.

Backports commit b4cd95d2f4c7197b844f51b29871d888063ea3e7 from qemu
2021-02-25 20:28:23 -05:00
Richard Henderson 2e03f74a53 target/arm: Use cpu_*_data_ra for sve_ldst_tlb_fn
Use the "normal" memory access functions, rather than the
softmmu internal helper functions directly.

Since fb901c9, cpu_mem_index is now a simple extract
from env->hflags and not a large computation.  Which means
that it's now more work to pass around this value than it
is to recompute it.

This only adjusts the primitives, and does not clean up
all of the uses within sve_helper.c.
2021-02-25 20:16:38 -05:00
Richard Henderson 5b3ddcf2e2 target/arm: Simplify DC_ZVA
Now that we know that the operation is on a single page,
we need not loop over pages while probing.

Backports commit e26d0d226892f67435cadcce86df0ddfb9943174 from qemu
2021-02-25 15:55:46 -05:00
Joseph Myers b08d204a37 softfloat: merge floatx80_mod and floatx80_rem
The m68k-specific softfloat code includes a function floatx80_mod that
is extremely similar to floatx80_rem, but computing the remainder
based on truncating the quotient toward zero rather than rounding it
to nearest integer. This is also useful for emulating the x87 fprem
and fprem1 instructions. Change the floatx80_rem implementation into
floatx80_modrem that can perform either operation, with both
floatx80_rem and floatx80_mod as thin wrappers available for all
targets.

There does not appear to be any use for the _mod operation for other
floating-point formats in QEMU (the only other architectures using
_rem at all are linux-user/arm/nwfpe, for FPA emulation, and openrisc,
for instructions that have been removed in the latest version of the
architecture), so no change is made to the code for other formats.

Backports commit 6b8b0136ab3018e4b552b485f808bf66bcf19ead from qemu
2021-02-25 13:34:05 -05:00
Alistair Francis e1f49dc888 target/riscv: Implement checks for hfence
Call the helper_hyp_tlb_flush() function on hfence instructions which
will generate an illegal insruction execption if we don't have
permission to flush the Hypervisor level TLBs.

Backports commit 2761db5fc20943bbd606b6fd49640ac000398de6 from qemu
2021-02-25 12:03:57 -05:00
Richard Henderson 1d95dd1c89 target/arm: Split helper_crypto_sm3tt
Rather than passing an opcode to a helper, fully decode the
operation at translate time. Use clear_tail_16 to zap the
balance of the SVE register with the AdvSIMD write.

Backports commit 43fa36c96c24349145497adc1b451f9caf74e344 from qemu
2020-06-14 23:24:21 -04:00
Richard Henderson 5ca8caf656 target/arm: Split helper_crypto_sha1_3reg
Rather than passing an opcode to a helper, fully decode the
operation at translate time. Use clear_tail_16 to zap the
balance of the SVE register with the AdvSIMD write.

Backports commit afc8b7d32668547308bdd654a63cf5228936e0ba from qemu
2020-06-14 23:18:45 -04:00
Richard Henderson 894f2168da target/arm: Convert rax1 to gvec helpers
With this conversion, we will be able to use the same helpers
with sve. This also fixes a bug in which we failed to clear
the high bits of the SVE register after an AdvSIMD operation.

Backports commit 1738860d7e60dec5dbeba17f8b44d31aae3accac from qemu
2020-06-14 22:49:36 -04:00
Richard Henderson cc3187b1e4 tcg: Implement gvec support for rotate by scalar
No host backend support yet, but the interfaces for rotls
are in place. Only implement left-rotate for now, as the
only known use of vector rotate by scalar is s390x, so any
right-rotate would be unused and untestable.

Backports commit 23850a74afb641102325b4b7f74071d929fc4594 from qemu
2020-06-14 22:00:50 -04:00
Richard Henderson be78062fd8 tcg: Implement gvec support for rotate by vector
No host backend support yet, but the interfaces for rotlv
and rotrv are in place.

Reviewed-by: Alex Bennée <alex.bennee@linaro.org>
Signed-off-by: Richard Henderson <richard.henderson@linaro.org>
---
v3: Drop the generic expansion from rot to shift; we can do better
for each backend, and then this code becomes unused.

Backports commit 5d0ceda902915e3f0e21c39d142c92c4e97c3ebb from qemu
2020-06-14 21:43:46 -04:00
Richard Henderson 5cce52a04b tcg: Implement gvec support for rotate by immediate
No host backend support yet, but the interfaces for rotli
are in place. Canonicalize immediate rotate to the left,
based on a survey of architectures, but provide both left
and right shift interfaces to the translators.

Backports commit b0f7e7444c03da17e41bf327c8aea590104a28ab from qemu
2020-06-14 21:26:58 -04:00
Peter Maydell bb0aa79847 target/arm: Convert Neon VADD, VSUB, VABD 3-reg-same insns to decodetree
Convert the Neon VADD, VSUB, VABD 3-reg-same insns to decodetree.
We already have gvec helpers for addition and subtraction, but must
add one for fabd.

Backports commit a26a352bb498662cd0c205cb433a352f86fac7d2 from qemu
2020-05-15 23:26:51 -04:00
Richard Henderson 451683ee79 target/arm: Vectorize SABA/UABA
Include 64-bit element size in preparation for SVE2.

Backports commit cfdb2c0c95ae9205b0dd7f0f5e970cdec50fef20 from qemu
2020-05-15 22:15:14 -04:00
Richard Henderson 5d7c46204d target/arm: Create gen_gvec_[us]sra
The functions eliminate duplication of the special cases for
this operation. They match up with the GVecGen2iFn typedef.

Add out-of-line helpers. We got away with only having inline
expanders because the neon vector size is only 16 bytes, and
we know that the inline expansion will always succeed.
When we reuse this for SVE, tcg-gvec-op may decide to use an
out-of-line helper due to longer vector lengths.

Backports commit 631e565450c483e0622eec3d8b61d7fa41d16bca from qemu
2020-05-15 20:10:32 -04:00
Richard Henderson 07f622e57d tcg: Add tcg_gen_gvec_dup_imm
Add a version of tcg_gen_dup_* that takes both immediate and
a vector element size operand. This will replace the set of
tcg_gen_gvec_dup{8,16,32,64}i functions that encode the element
size within the function name.

Backports commit 44c94677febd15488f9190b11eaa4a08e8ac696b from qemu
2020-05-07 09:55:25 -04:00
Thomas Huth 84f2729a29 target/arm: Make cpu_register() available for other files
Make cpu_register() (renamed to arm_cpu_register()) available
from internals.h so we can register CPUs also from other files
in the future.

Backports commit 37bcf244454f4efb82e2c0c64bbd7eabcc165a0c from qemu
2020-04-30 21:38:42 -04:00
Richard Henderson b26b4c06cd target/arm: Vectorize integer comparison vs zero
These instructions are often used in glibc's string routines.
They were the final uses of the 32-bit at a time neon helpers.

Backports commit 6b375d3546b009d1e63e07397ec9c6af256e15e9 from qemu
2020-04-30 21:29:17 -04:00
Anup Patel d47390ade4 target/riscv: Emulate TIME CSRs for privileged mode
Currently, TIME CSRs are emulated only for user-only mode. This
patch add TIME CSRs emulation for privileged mode.

For privileged mode, the TIME CSRs will return value provided
by rdtime callback which is registered by QEMU machine/platform
emulation (i.e. CLINT emulation). If rdtime callback is not
available then the monitor (i.e. OpenSBI) will trap-n-emulate
TIME CSRs in software.

We see 25+% performance improvement in hackbench numbers when
TIME CSRs are not trap-n-emulated.

Backports commit c695724868ce4049fd79c5a509880dbdf171e744 from qemu
2020-03-22 02:22:17 -04:00
Alistair Francis 347ad87e98 target/riscv: Disable guest FP support based on virtual status
When the Hypervisor extension is in use we only enable floating point
support when both status and vsstatus have enabled floating point
support.

Backports commit 29409c1d921d607873268671bf11a088efb5558e from qemu
2020-03-22 01:49:05 -04:00
Alistair Francis ebc7b9371f target/riscv: Add virtual register swapping function
Backports commit 66e594f2800ddc55f908830bf9e8dc4cda1304fe from qemu
2020-03-22 01:30:22 -04:00
Alistair Francis e4e41d3d87 target/riscv: Fix CSR perm checking for HS mode
Update the CSR permission checking to work correctly when we are in
HS-mode.

Backports commit 0a42f4c4408824dc7cb9ff60c9bdce6dcc0d24a5 from qemu
2020-03-22 01:19:04 -04:00
Alistair Francis 95537388c5 target/riscv: Add the virtulisation mode
Backports commit ef6bb7b62682badefdcb744831510aaa5971684f from qemu
2020-03-22 01:15:06 -04:00
Richard Henderson fcce8d4aa1 target/arm: Convert PMULL.8 to gvec
We still need two different helpers, since NEON and SVE2 get the
inputs from different locations within the source vector. However,
we can convert both to the same internal form for computation.

The sve2 helper is not used yet, but adding it with this patch
helps illustrate why the neon changes are helpful.

Backports commit e7e96fc5ec8c79dc77fef522d5226ac09f684ba5 from qemu
2020-03-21 19:35:46 -04:00
Richard Henderson c00f72f74f target/arm: Convert PMULL.64 to gvec
The gvec form will be needed for implementing SVE2.

Backports commit b9ed510e46f2f9e31e5e8adb4661d5d1cbe9a459 from qemu
2020-03-21 19:27:38 -04:00
Richard Henderson db8a935b44 target/arm: Convert PMUL.8 to gvec
The gvec form will be needed for implementing SVE2.

Extend the implementation to operate on uint64_t instead of uint32_t.
Use a counted inner loop instead of terminating when op1 goes to zero,
looking toward the required implementation for ARMv8.4-DIT.

Backports commit a21bb78e5817be3f494922e1dadd6455fe5d6318 from qemu
2020-03-21 19:22:18 -04:00
Richard Henderson d3139f2f0a target/arm: Vectorize USHL and SSHL
These instructions shift left or right depending on the sign
of the input, and 7 bits are significant to the shift. This
requires several masks and selects in addition to the actual
shifts to form the complete answer.

That said, the operation is still a small improvement even for
two 64-bit elements -- 13 vector operations instead of 2 * 7
integer operations.

Backports commit 87b74e8b6edd287ea2160caa0ebea725fa8f1ca1 from qemu
2020-03-21 19:14:17 -04:00
Richard Henderson 12b4e01d9c tcg: Add tcg_gen_gvec_5_ptr
Extend the vector generator infrastructure to handle
5 vector arguments.

Backports commit 2445971604c1cfd3ec484457159f4ac300fb04d2 from qemu
2020-03-21 16:54:01 -04:00
Richard Henderson d6150127b4 target/arm: Add the hypervisor virtual counter
Backports commit 8c94b071a09c2183f032febff3112f2b7662156c from qemu
2020-03-21 15:35:36 -04:00
Beata Michalska 0716794d86 Memory: Enable writeback for given memory region
Add an option to trigger memory writeback to sync given memory region
with the corresponding backing store, case one is available.
This extends the support for persistent memory, allowing syncing on-demand.

Backports commit 61c490e25e081af39ff40556f6c1229b8b011585 from qemu
2020-01-14 07:44:24 -05:00
Beata Michalska 47776dc862 tcg: cputlb: Add probe_read
Add probe_read alongside the write probing equivalent.

Backports commit 9e70492b4389d4355ae9c9ee2ba6286cfdadc257 from qemu
2020-01-14 07:16:41 -05:00
David Hildenbrand d9d91c1db6 tcg: Factor out probe_write() logic into probe_access()
Let's also allow to probe other access types.

Backports commit c25c283df0f08582df29f1d5d7be1516b851532d from qemu
2020-01-14 07:07:54 -05:00
Richard Henderson 07f30382c0 cputlb: Handle watchpoints via TLB_WATCHPOINT
The raising of exceptions from check_watchpoint, buried inside
of the I/O subsystem, is fundamentally broken. We do not have
the helper return address with which we can unwind guest state.

Replace PHYS_SECTION_WATCH and io_mem_watch with TLB_WATCHPOINT.
Move the call to cpu_check_watchpoint into the cputlb helpers
where we do have the helper return address.

This allows watchpoints on RAM to bypass the full i/o access path.

Backports commit 50b107c5d617eaf93301cef20221312e7a986701 from qemu
2020-01-14 06:58:33 -05:00
Tony Nguyen da98d0da4e memory: Access MemoryRegion with endianness
Preparation for collapsing the two byte swaps adjust_endianness and
handle_bswap into the former.

Call memory_region_dispatch_{read|write} with endianness encoded into
the "MemOp op" operand.

This patch does not change any behaviour as
memory_region_dispatch_{read|write} is yet to handle the endianness.

Once it does handle endianness, callers with byte swaps can collapse
them into adjust_endianness.

Backports commit d5d680cacc66ef7e3c02c81dc8f3a34eabce6dfe from qemu
2020-01-07 18:54:11 -05:00
Marc Zyngier 868de52f69 target/arm: Handle trapping to EL2 of AArch32 VMRS instructions
HCR_EL2.TID3 requires that AArch32 reads of MVFR[012] are trapped to
EL2, and HCR_EL2.TID0 does the same for reads of FPSID.
In order to handle this, introduce a new TCG helper function that
checks for these control bits before executing the VMRC instruction.

Tested with a hacked-up version of KVM/arm64 that sets the control
bits for 32bit guests.

Backports commit 9ca1d776cb49c09b09579d9edd0447542970c834 from qemu
2020-01-07 18:04:16 -05:00
Richard Henderson 3d3d56056b
target/arm: Remove helper_double_saturate
Replace x = double_saturate(y) with x = add_saturate(y, y).
There is no need for a separate more specialized helper.

Backports commit 640581a06d14e2d0d3c3ba79b916de6bc43578b0 from qemu
2019-11-18 20:13:21 -05:00
Michael Clark 6ab36ed89e
target/riscv: Implement riscv_cpu_unassigned_access
This patch adds support for the riscv_cpu_unassigned_access call
and will raise a load or store access fault.

Backports commit cbf5827693addaff4e4d2102afedbf078a204eb2 from qemu
2019-08-08 16:48:02 -04:00
Richard Henderson 2ea6dfbd63
tcg: Add support for vector compare select
Perform a per-element conditional move. This combination operation is
easier to implement on some host vector units than plain cmp+bitsel.
Omit the usual gvec interface, as this is intended to be used by
target-specific gvec expansion call-backs.

Backports commit f75da2988eb2457fa23d006d573220c5c680ec4e from qemu
2019-05-24 18:21:13 -04:00
Richard Henderson ca58be9cb4
tcg: Add support for vector bitwise select
This operation performs d = (b & a) | (c & ~a), and is present
on a majority of host vector units. Include gvec expanders.

Backports commit 38dc12947ec9106237f9cdbd428792c985cd86ae from qemu
2019-05-24 18:15:10 -04:00
Richard Henderson dab0061a0d
tcg: Use CPUClass::tlb_fill in cputlb.c
We can now use the CPUClass hook instead of a named function.

Create a static tlb_fill function to avoid other changes within
cputlb.c. This also isolates the asserts within. Remove the
named tlb_fill function from all of the targets.

Backports commit c319dc13579a92937bffe02ad2c9f1a550e73973 from qemu
2019-05-16 17:35:37 -04:00
Richard Henderson e98c731550
target/riscv: Convert to CPUClass::tlb_fill
Note that env->pc is removed from the qemu_log as that value is garbage.
The PC isn't recovered until cpu_restore_state, called from
cpu_loop_exit_restore, called from riscv_raise_exception.

Backports commit 8a4ca3c10a96be6ed7f023b685b688c4d409bbcb from qemu
2019-05-16 17:24:01 -04:00
Richard Henderson 31ecdb5341
target/arm: Convert to CPUClass::tlb_fill
Backports commit 7350d553b5066abdc662045d7db5cdb73d0f9d53 from qemu
2019-05-16 16:55:12 -04:00
Richard Henderson 552e48f14e
target/arm: Use tcg_gen_abs_i64 and tcg_gen_gvec_abs
Backports commit 4e027a710673f5d4dc6cff88728bcfd32e4c47b0 from qemu
2019-05-16 16:43:02 -04:00
Richard Henderson 6d5e7856ff
tcg: Add support for vector absolute value
Backports commit bcefc90208f8a1d6f619d61c2647281d92277015 from qemu
2019-05-16 16:33:43 -04:00
Richard Henderson 6d1730048d
tcg: Add support for integer absolute value
Remove a function of the same name from target/arm/.
Use a branchless implementation of abs gleaned from gcc.

Backports commit ff1f11f7f8710a768f9313f24bd7f509d3db27e5 from qemu
2019-05-16 16:25:15 -04:00
Richard Henderson 79b9dc559e
tcg: Add gvec expanders for vector shift by scalar
Allow expansion either via shift by scalar or by replicating
the scalar for shift by vector.

Backports commit b4578cd91cda4cef1c413304353ca6dc5b957b60 from qemu
2019-05-16 16:17:58 -04:00
Richard Henderson 8c17687934
tcg: Add gvec expanders for variable shift
The gvec expanders perform a modulo on the shift count. If the target
requires alternate behaviour, then it cannot use the generic gvec
expanders anyway, and will have to have its own custom code.

Backports commit 5ee5c14cacda27e904cd6b0d9e7ffe1acff42838 from qemu
2019-05-16 15:51:09 -04:00
Richard Henderson 66e6bea084
tcg: Add INDEX_op_dupm_vec
Allow the backend to expand dup from memory directly, instead of
forcing the value into a temp first. This is especially important
if integer/vector register moves do not exist.

Note that officially tcg_out_dupm_vec is allowed to fail.
If it did, we could fix this up relatively easily:

VECE == 32/64:
Load the value into a vector register, then dup.
Both of these must work.

VECE == 8/16:
If the value happens to be at an offset such that an aligned
load would place the desired value in the least significant
end of the register, go ahead and load w/garbage in high bits.

Load the value w/INDEX_op_ld{8,16}_i32.
Attempt a move directly to vector reg, which may fail.
Store the value into the backing store for OTS.
Load the value into the vector reg w/TCG_TYPE_I32, which must work.
Duplicate from the vector reg into itself, which must work.

All of which is well and good, except that all supported
hosts can support dupm for all vece, so all of the failure
paths would be dead code and untestable.

Backports commit 37ee55a081b7863ffab2151068dd1b2f11376914 from qemu
2019-05-16 15:38:02 -04:00
Richard Henderson c54b2776f6
tcg: Specify optional vector requirements with a list
Replace the single opcode in .opc with a null-terminated
array in .opt_opc. We still require that all opcodes be
used with the same .vece.

Validate the contents of this list with CONFIG_DEBUG_TCG.
All tcg_gen_*_vec functions will check any list active
during .fniv expansion. Swap the active list in and out
as we expand other opcodes, or take control away from the
front-end function.

Convert all existing vector aware front ends.

Backports commit 53229a7703eeb2bbe101a19a33ef22aaf960c65b from qemu
2019-05-16 15:05:02 -04:00
David Hildenbrand f3b4a64d27
tcg: Implement tcg_gen_gvec_3i()
Let's add tcg_gen_gvec_3i(), similar to tcg_gen_gvec_2i(), however
without introducing "gen_helper_gvec_3i *fnoi", as it isn't needed
for now.

Backports commit e1227bb6e59173117f094a6a13b998587b45c928 from qemu
2019-05-16 14:26:50 -04:00
Peter Maydell 77ae3982b4
target/arm: Implement VLLDM for v7M CPUs with an FPU
Implement the VLLDM instruction for v7M for the FPU present cas.

Backports commit 956fe143b4f254356496a0a1c479fa632376dfec from qemu
2019-04-30 11:27:54 -04:00