Commit graph

434 commits

Author SHA1 Message Date
MerryMage f9b909f529 header_gen: Add gen_{u,s}shl_i{32,64} to arm 2020-04-13 19:38:59 +01:00
Anup Patel d47390ade4 target/riscv: Emulate TIME CSRs for privileged mode
Currently, TIME CSRs are emulated only for user-only mode. This
patch add TIME CSRs emulation for privileged mode.

For privileged mode, the TIME CSRs will return value provided
by rdtime callback which is registered by QEMU machine/platform
emulation (i.e. CLINT emulation). If rdtime callback is not
available then the monitor (i.e. OpenSBI) will trap-n-emulate
TIME CSRs in software.

We see 25+% performance improvement in hackbench numbers when
TIME CSRs are not trap-n-emulated.

Backports commit c695724868ce4049fd79c5a509880dbdf171e744 from qemu
2020-03-22 02:22:17 -04:00
Alistair Francis 347ad87e98 target/riscv: Disable guest FP support based on virtual status
When the Hypervisor extension is in use we only enable floating point
support when both status and vsstatus have enabled floating point
support.

Backports commit 29409c1d921d607873268671bf11a088efb5558e from qemu
2020-03-22 01:49:05 -04:00
Alistair Francis ebc7b9371f target/riscv: Add virtual register swapping function
Backports commit 66e594f2800ddc55f908830bf9e8dc4cda1304fe from qemu
2020-03-22 01:30:22 -04:00
Alistair Francis ab7c1570d8 target/riscv: Add the force HS exception mode
Add a FORCE_HS_EXCEP mode to the RISC-V virtulisation status. This bit
specifies if an exeption should be taken to HS mode no matter the
current delegation status. This is used when an exeption must be taken
to HS mode, such as when handling interrupts.

Backports commit c7b1bbc80fc2af17395d3986c346fd2307e57829 from qemu
2020-03-22 01:16:55 -04:00
Alistair Francis 95537388c5 target/riscv: Add the virtulisation mode
Backports commit ef6bb7b62682badefdcb744831510aaa5971684f from qemu
2020-03-22 01:15:06 -04:00
Richard Henderson fcce8d4aa1 target/arm: Convert PMULL.8 to gvec
We still need two different helpers, since NEON and SVE2 get the
inputs from different locations within the source vector. However,
we can convert both to the same internal form for computation.

The sve2 helper is not used yet, but adding it with this patch
helps illustrate why the neon changes are helpful.

Backports commit e7e96fc5ec8c79dc77fef522d5226ac09f684ba5 from qemu
2020-03-21 19:35:46 -04:00
Richard Henderson c00f72f74f target/arm: Convert PMULL.64 to gvec
The gvec form will be needed for implementing SVE2.

Backports commit b9ed510e46f2f9e31e5e8adb4661d5d1cbe9a459 from qemu
2020-03-21 19:27:38 -04:00
Richard Henderson db8a935b44 target/arm: Convert PMUL.8 to gvec
The gvec form will be needed for implementing SVE2.

Extend the implementation to operate on uint64_t instead of uint32_t.
Use a counted inner loop instead of terminating when op1 goes to zero,
looking toward the required implementation for ARMv8.4-DIT.

Backports commit a21bb78e5817be3f494922e1dadd6455fe5d6318 from qemu
2020-03-21 19:22:18 -04:00
Richard Henderson d3139f2f0a target/arm: Vectorize USHL and SSHL
These instructions shift left or right depending on the sign
of the input, and 7 bits are significant to the shift. This
requires several masks and selects in addition to the actual
shifts to form the complete answer.

That said, the operation is still a small improvement even for
two 64-bit elements -- 13 vector operations instead of 2 * 7
integer operations.

Backports commit 87b74e8b6edd287ea2160caa0ebea725fa8f1ca1 from qemu
2020-03-21 19:14:17 -04:00
Richard Henderson 12b4e01d9c tcg: Add tcg_gen_gvec_5_ptr
Extend the vector generator infrastructure to handle
5 vector arguments.

Backports commit 2445971604c1cfd3ec484457159f4ac300fb04d2 from qemu
2020-03-21 16:54:01 -04:00
Richard Henderson d6150127b4 target/arm: Add the hypervisor virtual counter
Backports commit 8c94b071a09c2183f032febff3112f2b7662156c from qemu
2020-03-21 15:35:36 -04:00
Richard Henderson 3a5135473f target/arm: Update arm_sctlr for VHE
Use the correct sctlr for EL2&0 regime. Due to header ordering,
and where arm_mmu_idx_el is declared, we need to move the function
out of line. Use the function in many more places in order to
select the correct control.

Backports commit aaec143212bb70ac9549cf73203d13100bd5c7c2 from qemu
2020-03-21 15:29:21 -04:00
Richard Henderson 3d583cd45f target/arm: Split out arm_mmu_idx_el
Backports commit 164690b29f9eaf69fe641859bc9f8954f12e691d from qemu
2020-03-21 15:22:01 -04:00
Richard Henderson 0318d7af99 target/arm: Reorganize ARMMMUIdx
Prepare for, but do not yet implement, the EL2&0 regime.
This involves adding the new MMUIdx enumerators and adjusting
some of the MMUIdx related predicates to match.

Backports commit b9f6033c1a5fb7da55ed353794db8ec064f78bb2 from qemu.
2020-03-21 15:10:05 -04:00
Yongbok Kim 7fbc373f59 target/mips: Add implementation of GINVT instruction
Implement emulation of GINVT instruction. As QEMU doesn't support
caches and virtualization, this implementation covers only one
instruction (GINVT - Global Invalidate TLB) among all TLB-related
MIPS instructions.

Backports commit 99029be1c2875cd857614397674bbf563ddb6f91 from qemu
2020-03-21 13:01:35 -04:00
Yongbok Kim f10de71e73 target/mips: Amend CP0 WatchHi register implementation
WatchHi is extended by the field MemoryMapID with the GINVT instruction.
The field is accessible by MTHC0/MFHC0 in 32-bit architectures and DMTC0/
DMFC0 in 64-bit architectures.

Backports commit feafe82cc2289a31b3e3f11dc76f3539ea22d670 from qemu
2020-03-21 12:39:00 -04:00
Beata Michalska 0716794d86 Memory: Enable writeback for given memory region
Add an option to trigger memory writeback to sync given memory region
with the corresponding backing store, case one is available.
This extends the support for persistent memory, allowing syncing on-demand.

Backports commit 61c490e25e081af39ff40556f6c1229b8b011585 from qemu
2020-01-14 07:44:24 -05:00
Beata Michalska 47776dc862 tcg: cputlb: Add probe_read
Add probe_read alongside the write probing equivalent.

Backports commit 9e70492b4389d4355ae9c9ee2ba6286cfdadc257 from qemu
2020-01-14 07:16:41 -05:00
David Hildenbrand d9d91c1db6 tcg: Factor out probe_write() logic into probe_access()
Let's also allow to probe other access types.

Backports commit c25c283df0f08582df29f1d5d7be1516b851532d from qemu
2020-01-14 07:07:54 -05:00
Richard Henderson 07f30382c0 cputlb: Handle watchpoints via TLB_WATCHPOINT
The raising of exceptions from check_watchpoint, buried inside
of the I/O subsystem, is fundamentally broken. We do not have
the helper return address with which we can unwind guest state.

Replace PHYS_SECTION_WATCH and io_mem_watch with TLB_WATCHPOINT.
Move the call to cpu_check_watchpoint into the cputlb helpers
where we do have the helper return address.

This allows watchpoints on RAM to bypass the full i/o access path.

Backports commit 50b107c5d617eaf93301cef20221312e7a986701 from qemu
2020-01-14 06:58:33 -05:00
Tony Nguyen da98d0da4e memory: Access MemoryRegion with endianness
Preparation for collapsing the two byte swaps adjust_endianness and
handle_bswap into the former.

Call memory_region_dispatch_{read|write} with endianness encoded into
the "MemOp op" operand.

This patch does not change any behaviour as
memory_region_dispatch_{read|write} is yet to handle the endianness.

Once it does handle endianness, callers with byte swaps can collapse
them into adjust_endianness.

Backports commit d5d680cacc66ef7e3c02c81dc8f3a34eabce6dfe from qemu
2020-01-07 18:54:11 -05:00
Marc Zyngier 868de52f69 target/arm: Handle trapping to EL2 of AArch32 VMRS instructions
HCR_EL2.TID3 requires that AArch32 reads of MVFR[012] are trapped to
EL2, and HCR_EL2.TID0 does the same for reads of FPSID.
In order to handle this, introduce a new TCG helper function that
checks for these control bits before executing the VMRC instruction.

Tested with a hacked-up version of KVM/arm64 that sets the control
bits for 32bit guests.

Backports commit 9ca1d776cb49c09b09579d9edd0447542970c834 from qemu
2020-01-07 18:04:16 -05:00
Peter Maydell 77d90985cc
target/sparc: Switch to do_transaction_failed() hook
Switch the SPARC target from the old unassigned_access hook to the
new do_transaction_failed hook.

This will cause the "if transaction failed" code paths added in
the previous commits to become active if the access is to an
unassigned address. In particular we'll now handle bus errors
during page table walks correctly (generating a translation
error with the right kind of fault status).

Backports commit f8c3db33a5e863291182f8862ddf81618a7c6194 from qemu
2019-11-28 02:56:50 -05:00
Peter Maydell 2faffb5af1
target/mips: Switch to do_transaction_failed() hook
Switch the MIPS target from the old unassigned_access hook to the new
do_transaction_failed hook.

Unlike the old hook, do_transaction_failed is only ever called from
the TCG memory access paths, so there is no need for the "ignore this
if we're using KVM" hack that we were previously using to work around
the way unassigned_access was called for all kinds of memory accesses
to unassigned physical addresses.

The MIPS target does not ever do direct memory reads by physical
address (via either ldl_phys etc or address_space_ldl etc), so the
only memory accesses this affects are the 'normal' guest loads and
stores, which will be handled by the new hook; their behaviour is
unchanged.

Backports commit 4f02a06d50ef0081089ed8cb3ec7c7986e3c95f8 from qemu
2019-11-28 02:54:53 -05:00
Richard Henderson 1f59a43544
Revert "target/arm: Use unallocated_encoding for aarch32"
Despite the fact that the text for the call to gen_exception_insn
is identical for aarch64 and aarch32, the implementation inside
gen_exception_insn is totally different.

This fixes exceptions raised from aarch64.

This reverts commit fb2d3c9a9a.
2019-11-18 23:49:47 -05:00
Richard Henderson 3d3d56056b
target/arm: Remove helper_double_saturate
Replace x = double_saturate(y) with x = add_saturate(y, y).
There is no need for a separate more specialized helper.

Backports commit 640581a06d14e2d0d3c3ba79b916de6bc43578b0 from qemu
2019-11-18 20:13:21 -05:00
Richard Henderson fb2d3c9a9a
target/arm: Use unallocated_encoding for aarch32
Promote this function from aarch64 to fully general use.
Use it to unify the code sequences for generating illegal
opcode exceptions.

Backports commit 3cb36637157088892e9e33ddb1034bffd1251d3b from qemu
2019-11-18 20:10:50 -05:00
Michael Clark 6ab36ed89e
target/riscv: Implement riscv_cpu_unassigned_access
This patch adds support for the riscv_cpu_unassigned_access call
and will raise a load or store access fault.

Backports commit cbf5827693addaff4e4d2102afedbf078a204eb2 from qemu
2019-08-08 16:48:02 -04:00
Philippe Mathieu-Daudé fa2a772c7b
target/arm: Declare some M-profile functions publicly
In the next commit we will split the M-profile functions from this
file. Some function will be called out of helper.c. Declare them in
the "internals.h" header.

Backports commit 787a7e76c2e93a48c47b324fea592c9910a70483 from qemu
2019-08-08 15:37:01 -04:00
Philippe Mathieu-Daudé 91e264823e
target/arm: Move TLB related routines to tlb_helper.c
These routines are TCG specific.
The arm_deliver_fault() function is only used within the new
helper. Make it static.

Backports commit e21b551cb652663f2f2405a64d63ef6b4a1042b7 from qemu
2019-08-08 15:24:26 -04:00
Philippe Mathieu-Daudé 1af5deaf52
target/arm: Declare get_phys_addr() function publicly
In the next commit we will split the TLB related routines of
this file, and this function will also be called in the new
file. Declare it in the "internals.h" header.

Backports commit ebae861fc6c385a7bcac72dde4716be06e6776f1 from qemu
2019-08-08 15:14:45 -04:00
Lioncash 9bf229ede1
header_gen: Add missing vfp_expand_imm entry to arm_symbols
Prevents multiple definition errors when building 32-bit ARM frontends.
2019-08-08 15:07:21 -04:00
Richard Henderson 7c32498b7f
target/arm: Use tcg_gen_gvec_bitsel
This replaces 3 target-specific implementations for BIT, BIF, and BSL.

Backports commit 3a7a2b4e5cf0d49cd8b14e8225af0310068b7d20 from qemu
2019-06-13 16:12:56 -04:00
Mateja Marjanovic 9e8aed043e
target/mips: Refactor and fix INSERT.<B|H|W|D> instructions
The old version of the helper for the INSERT.<B|H|W|D> MSA instructions
has been replaced with four helpers that don't use switch, and change
the endianness of the given index, when executed on a big endian host.

Backports commit c1c9a10fb1f7a6782711817c167a2c20b000fc12 from qemu
2019-05-28 19:42:28 -04:00
Mateja Marjanovic d6a8d25015
target/mips: Refactor and fix COPY_U.<B|H|W> instructions
The old version of the helper for the COPY_U.<B|H|W> MSA instructions
has been replaced with four helpers that don't use switch, and change
the endianness of the given index, when executed on a big endian host.

Backports commit 41d288582782cf8d63241ecb6efa1e4160fe78f7 from qemu
2019-05-28 19:39:22 -04:00
Mateja Marjanovic 54a33d1db3
target/mips: Refactor and fix COPY_S.<B|H|W|D> instructions
The old version of the helper for the COPY_S.<B|H|W|D> MSA instructions
has been replaced with four helpers that don't use switch, and change
the endianness of the given index, when executed on a big endian host.

Backports commit 631c467461496dcf6d6a3e4c3d27a1433e96868e from qemu
2019-05-28 19:36:14 -04:00
Richard Henderson 2ea6dfbd63
tcg: Add support for vector compare select
Perform a per-element conditional move. This combination operation is
easier to implement on some host vector units than plain cmp+bitsel.
Omit the usual gvec interface, as this is intended to be used by
target-specific gvec expansion call-backs.

Backports commit f75da2988eb2457fa23d006d573220c5c680ec4e from qemu
2019-05-24 18:21:13 -04:00
Richard Henderson ca58be9cb4
tcg: Add support for vector bitwise select
This operation performs d = (b & a) | (c & ~a), and is present
on a majority of host vector units. Include gvec expanders.

Backports commit 38dc12947ec9106237f9cdbd428792c985cd86ae from qemu
2019-05-24 18:15:10 -04:00
Richard Henderson dab0061a0d
tcg: Use CPUClass::tlb_fill in cputlb.c
We can now use the CPUClass hook instead of a named function.

Create a static tlb_fill function to avoid other changes within
cputlb.c. This also isolates the asserts within. Remove the
named tlb_fill function from all of the targets.

Backports commit c319dc13579a92937bffe02ad2c9f1a550e73973 from qemu
2019-05-16 17:35:37 -04:00
Richard Henderson 5d83199931
target/sparc: Convert to CPUClass::tlb_fill
Backports commit e84942f2ceaa79430414f2cb68d77c044dadca96 from qemu
2019-05-16 17:29:35 -04:00
Richard Henderson e98c731550
target/riscv: Convert to CPUClass::tlb_fill
Note that env->pc is removed from the qemu_log as that value is garbage.
The PC isn't recovered until cpu_restore_state, called from
cpu_loop_exit_restore, called from riscv_raise_exception.

Backports commit 8a4ca3c10a96be6ed7f023b685b688c4d409bbcb from qemu
2019-05-16 17:24:01 -04:00
Richard Henderson 14d48974a4
target/mips: Convert to CPUClass::tlb_fill
Note that env->active_tc.PC is removed from the qemu_log as that value
is garbage. The PC isn't recovered until cpu_restore_state, called from
cpu_loop_exit_restore, called from do_raise_exception_err.

Backports commit 931d019f5b2e7bbacb162869497123be402ddd86 from qemu
2019-05-16 17:19:47 -04:00
Richard Henderson 31ecdb5341
target/arm: Convert to CPUClass::tlb_fill
Backports commit 7350d553b5066abdc662045d7db5cdb73d0f9d53 from qemu
2019-05-16 16:55:12 -04:00
Richard Henderson 552e48f14e
target/arm: Use tcg_gen_abs_i64 and tcg_gen_gvec_abs
Backports commit 4e027a710673f5d4dc6cff88728bcfd32e4c47b0 from qemu
2019-05-16 16:43:02 -04:00
Richard Henderson 6d5e7856ff
tcg: Add support for vector absolute value
Backports commit bcefc90208f8a1d6f619d61c2647281d92277015 from qemu
2019-05-16 16:33:43 -04:00
Richard Henderson 6d1730048d
tcg: Add support for integer absolute value
Remove a function of the same name from target/arm/.
Use a branchless implementation of abs gleaned from gcc.

Backports commit ff1f11f7f8710a768f9313f24bd7f509d3db27e5 from qemu
2019-05-16 16:25:15 -04:00
Richard Henderson 79b9dc559e
tcg: Add gvec expanders for vector shift by scalar
Allow expansion either via shift by scalar or by replicating
the scalar for shift by vector.

Backports commit b4578cd91cda4cef1c413304353ca6dc5b957b60 from qemu
2019-05-16 16:17:58 -04:00
Richard Henderson 8c17687934
tcg: Add gvec expanders for variable shift
The gvec expanders perform a modulo on the shift count. If the target
requires alternate behaviour, then it cannot use the generic gvec
expanders anyway, and will have to have its own custom code.

Backports commit 5ee5c14cacda27e904cd6b0d9e7ffe1acff42838 from qemu
2019-05-16 15:51:09 -04:00
Richard Henderson 66e6bea084
tcg: Add INDEX_op_dupm_vec
Allow the backend to expand dup from memory directly, instead of
forcing the value into a temp first. This is especially important
if integer/vector register moves do not exist.

Note that officially tcg_out_dupm_vec is allowed to fail.
If it did, we could fix this up relatively easily:

VECE == 32/64:
Load the value into a vector register, then dup.
Both of these must work.

VECE == 8/16:
If the value happens to be at an offset such that an aligned
load would place the desired value in the least significant
end of the register, go ahead and load w/garbage in high bits.

Load the value w/INDEX_op_ld{8,16}_i32.
Attempt a move directly to vector reg, which may fail.
Store the value into the backing store for OTS.
Load the value into the vector reg w/TCG_TYPE_I32, which must work.
Duplicate from the vector reg into itself, which must work.

All of which is well and good, except that all supported
hosts can support dupm for all vece, so all of the failure
paths would be dead code and untestable.

Backports commit 37ee55a081b7863ffab2151068dd1b2f11376914 from qemu
2019-05-16 15:38:02 -04:00