Commit graph

486 commits

Author SHA1 Message Date
LIU Zhiwei 244793c4e8 target/riscv: vector single-width bit shift instructions
Backports 3277d955d21d8943d80062b4cfd8547f831dbd51
2021-02-26 10:37:09 -05:00
LIU Zhiwei 56c0e253c2 target/riscv: vector bitwise logical instructions
Backports d3842924cf93d104f691c5ea9090d6700ccef281
2021-02-26 10:30:33 -05:00
LIU Zhiwei 05153c6d7c target/riscv: vector integer add-with-carry / subtract-with-borrow instructions
3a6f8f68ad2f4a22d9ae8287f336b5dcc80b6448
2021-02-26 10:19:48 -05:00
LIU Zhiwei b9814de4c3 target/riscv: vector widening integer add and subtract
Backports 8fcdf77630290591a6068c2d82ca2935338c3b0c
2021-02-26 10:05:43 -05:00
LIU Zhiwei f564388e89 target/riscv: vector single-width integer add and subtract
Backports 43740e3a3b3bb66456103684e622ba4e9baae297
2021-02-26 09:58:31 -05:00
LIU Zhiwei 7d0d7338c2 target/riscv: add vector amo operations
Vector AMOs operate as if aq and rl bits were zero on each element
with regard to ordering relative to other instructions in the same hart.
Vector AMOs provide no ordering guarantee between element operations
in the same vector AMO instruction

Backports 268fcca66bde62257960ec8d859de374315a5e3d
2021-02-26 09:47:32 -05:00
LIU Zhiwei 152934bade target/riscv: add fault-only-first unit stride load
The unit-stride fault-only-fault load instructions are used to
vectorize loops with data-dependent exit conditions(while loops).
These instructions execute as a regular load except that they
will only take a trap on element 0.

Backports commit 022b4ecf775ffeff522eaea4f0d94edcfe00a0a9 from qemu
2021-02-26 09:28:19 -05:00
LIU Zhiwei 887c29bc79 target/riscv: add vector index load and store instructions
Vector indexed operations add the contents of each element of the
vector offset operand specified by vs2 to the base effective address
to give the effective address of each element.

Backports f732560e3551c0823cee52efba993fbb8f689a36
2021-02-26 03:00:45 -05:00
LIU Zhiwei c7a17d04a2 target/riscv: add vector stride load and store instructions
Vector strided operations access the first memory element at the base address,
and then access subsequent elements at address increments given by the byte
offset contained in the x register specified by rs2.

Vector unit-stride operations access elements stored contiguously in memory
starting from the base effective address. It can been seen as a special
case of strided operations.

Backports 751538d5da557e5c10e5045c2d27639580ea54a7
2021-02-26 02:55:14 -05:00
LIU Zhiwei 9db3b70869 target/riscv: add vector configure instruction
vsetvl and vsetvli are two configure instructions for vl, vtype. TB flags
should update after configure instructions. The (ill, lmul, sew ) of vtype
and the bit of (VSTART == 0 && VL == VLMAX) will be placed within tb_flags.

Backports 2b7168fc43fb270fb89e1dddc17ef54714712f3a from qemu
2021-02-26 02:37:59 -05:00
Richard Henderson 57c66389c2 target/arm: Fix temp double-free in sve ldr/str
The temp that gets assigned to clean_addr has been allocated with
new_tmp_a64, which means that it will be freed at the end of the
instruction. Freeing it earlier leads to assertion failure.

The loop creates a complication, in which we allocate a new local
temp, which does need freeing, and the final code path is shared
between the loop and non-loop.

Fix this complication by adding new_tmp_a64_local so that the new
local temp is freed at the end, and can be treated exactly like
the non-loop path.

Fixes: bba87d0a0f4

Backports commit 4b4dc9750a0aa0b9766bd755bf6512a84744ce8a from qemu
2021-02-25 23:10:37 -05:00
Richard Henderson 732efce958 target/arm: Add mte helpers for sve scatter/gather memory ops
Because the elements are non-sequential, we cannot eliminate many
tests straight away like we can for sequential operations. But
we often have the PTE details handy, so we can test for Tagged.

Backports commit d28d12f008ee44dc2cc2ee5d8f673be9febc951e from qemu
2021-02-25 22:34:24 -05:00
Richard Henderson 5698b7badb target/arm: Handle TBI for sve scalar + int memory ops
We still need to handle tbi for user-only when mte is inactive.

Backports commit 9473d0ecafcffc8b258892b1f9f18e037bdba958 from qemu
2021-02-25 22:17:46 -05:00
Richard Henderson 586235d02d target/arm: Add mte helpers for sve scalar + int ff/nf loads
Because the elements are sequential, we can eliminate many tests all
at once when the tag hits TCMA, or if the page(s) are not Tagged.

Backports commit aa13f7c3c378fa41366b9fcd6c29af1c3d81126a from qemu
2021-02-25 22:09:17 -05:00
Richard Henderson cb31d54b18 target/arm: Add mte helpers for sve scalar + int stores
Because the elements are sequential, we can eliminate many tests all
at once when the tag hits TCMA, or if the page(s) are not Tagged.

Backports commit 71b9f3948c75bb97641a3c8c7de96d1cb47cdc07 from qemu
2021-02-25 21:53:55 -05:00
Richard Henderson 670b25c5fa target/arm: Add mte helpers for sve scalar + int loads
Because the elements are sequential, we can eliminate many tests all
at once when the tag hits TCMA, or if the page(s) are not Tagged.

Backports commit 206adacfb8d35e671e3619591608c475aa046b63 from qemu
2021-02-25 21:45:32 -05:00
Richard Henderson 94b0876f15 target/arm: Add sve infrastructure for page lookup
For contiguous predicated memory operations, we want to
minimize the number of tlb lookups performed. We have
open-coded this for sve_ld1_r, but for correctness with
MTE we will need this for all of the memory operations.

Create a structure that holds the bounds of active elements,
and metadata for two pages. Add routines to find those
active elements, lookup the pages, and run watchpoints
for those pages.

Temporarily mark the functions unused to avoid Werror.

Backports commit b4cd95d2f4c7197b844f51b29871d888063ea3e7 from qemu
2021-02-25 20:28:23 -05:00
Richard Henderson 2e03f74a53 target/arm: Use cpu_*_data_ra for sve_ldst_tlb_fn
Use the "normal" memory access functions, rather than the
softmmu internal helper functions directly.

Since fb901c9, cpu_mem_index is now a simple extract
from env->hflags and not a large computation.  Which means
that it's now more work to pass around this value than it
is to recompute it.

This only adjusts the primitives, and does not clean up
all of the uses within sve_helper.c.
2021-02-25 20:16:38 -05:00
Richard Henderson 49bd9a5c68 target/arm: Use mte_checkN for sve unpredicated stores
Backports commit bba87d0a0f480805223a6428a7942a51733c488a from qemu
2021-02-25 17:40:43 -05:00
Richard Henderson 4fdd05e1aa target/arm: Add helper_mte_check_zva
Use a special helper for DC_ZVA, rather than the more
general mte_checkN.

Backports commit 46dc1bc0601554823a42ad27f236da2ad8f3bdc6 from qemu
2021-02-25 17:17:54 -05:00
Richard Henderson 9a05ca01e7 target/arm: Implement helper_mte_checkN
Fill out the stub that was added earlier.

Backports commit 5add8248556a3c1006018d7d8e601c9572b280a9 from qemu
2021-02-25 17:10:56 -05:00
Richard Henderson 91e2f55b69 target/arm: Implement helper_mte_check1
Fill out the stub that was added earlier.

Backports commit 2e34ff45f32cb032883616a1cc5ea8ac96f546d5 from qemu
2021-02-25 17:02:34 -05:00
Richard Henderson 3e786526cf target/arm: Add gen_mte_checkN
Replace existing uses of check_data_tbi in translate-a64.c that
perform multiple logical memory access. Leave the helper blank
for now to reduce the patch size.

Backports commit 73ceeb0011b23bac8bd2c09ebe3c18d034aa69ce from qemu
2021-02-25 16:40:16 -05:00
Richard Henderson 582e64f348 target/arm: Add gen_mte_check1
Replace existing uses of check_data_tbi in translate-a64.c that
perform a single logical memory access. Leave the helper blank
for now to reduce the patch size.

Backports commit 0a405be2b8fd9506a009b10d7d2d98c394b36db6 from qemu
2021-02-25 16:13:31 -05:00
Richard Henderson 4bb37fc3c1 target/arm: Implement the LDGM, STGM, STZGM instructions
Backports commit 5f716a82388eb09754dd900e7dbb8ffa15897a28 from qemu
2021-02-25 16:00:50 -05:00
Richard Henderson 5b3ddcf2e2 target/arm: Simplify DC_ZVA
Now that we know that the operation is on a single page,
we need not loop over pages while probing.

Backports commit e26d0d226892f67435cadcce86df0ddfb9943174 from qemu
2021-02-25 15:55:46 -05:00
Richard Henderson e8b9cb8b4a target/arm: Implement LDG, STG, ST2G instructions
Backports commit c15294c1e36a7dd9b25bd54d98178e80f4b64bc1 from qemu
2021-02-25 15:08:44 -05:00
Richard Henderson 911a6b57ed target/arm: Implement the ADDG, SUBG instructions
Backports commit efbc78ad978763aedd11cb718eb1ff8db3fc9152 from qemu
2021-02-25 14:42:33 -05:00
Richard Henderson 58f3dd2cc7 target/arm: Implement the IRG instruction
Backports commit da54941f45b820cbaca72aa6efd5669b3dc86e2f from qemu
2021-02-25 14:36:11 -05:00
Joseph Myers b08d204a37 softfloat: merge floatx80_mod and floatx80_rem
The m68k-specific softfloat code includes a function floatx80_mod that
is extremely similar to floatx80_rem, but computing the remainder
based on truncating the quotient toward zero rather than rounding it
to nearest integer. This is also useful for emulating the x87 fprem
and fprem1 instructions. Change the floatx80_rem implementation into
floatx80_modrem that can perform either operation, with both
floatx80_rem and floatx80_mod as thin wrappers available for all
targets.

There does not appear to be any use for the _mod operation for other
floating-point formats in QEMU (the only other architectures using
_rem at all are linux-user/arm/nwfpe, for FPA emulation, and openrisc,
for instructions that have been removed in the latest version of the
architecture), so no change is made to the code for other formats.

Backports commit 6b8b0136ab3018e4b552b485f808bf66bcf19ead from qemu
2021-02-25 13:34:05 -05:00
Alistair Francis e1f49dc888 target/riscv: Implement checks for hfence
Call the helper_hyp_tlb_flush() function on hfence instructions which
will generate an illegal insruction execption if we don't have
permission to flush the Hypervisor level TLBs.

Backports commit 2761db5fc20943bbd606b6fd49640ac000398de6 from qemu
2021-02-25 12:03:57 -05:00
Richard Henderson 1d95dd1c89 target/arm: Split helper_crypto_sm3tt
Rather than passing an opcode to a helper, fully decode the
operation at translate time. Use clear_tail_16 to zap the
balance of the SVE register with the AdvSIMD write.

Backports commit 43fa36c96c24349145497adc1b451f9caf74e344 from qemu
2020-06-14 23:24:21 -04:00
Richard Henderson 5ca8caf656 target/arm: Split helper_crypto_sha1_3reg
Rather than passing an opcode to a helper, fully decode the
operation at translate time. Use clear_tail_16 to zap the
balance of the SVE register with the AdvSIMD write.

Backports commit afc8b7d32668547308bdd654a63cf5228936e0ba from qemu
2020-06-14 23:18:45 -04:00
Richard Henderson 894f2168da target/arm: Convert rax1 to gvec helpers
With this conversion, we will be able to use the same helpers
with sve. This also fixes a bug in which we failed to clear
the high bits of the SVE register after an AdvSIMD operation.

Backports commit 1738860d7e60dec5dbeba17f8b44d31aae3accac from qemu
2020-06-14 22:49:36 -04:00
Richard Henderson cc3187b1e4 tcg: Implement gvec support for rotate by scalar
No host backend support yet, but the interfaces for rotls
are in place. Only implement left-rotate for now, as the
only known use of vector rotate by scalar is s390x, so any
right-rotate would be unused and untestable.

Backports commit 23850a74afb641102325b4b7f74071d929fc4594 from qemu
2020-06-14 22:00:50 -04:00
Richard Henderson be78062fd8 tcg: Implement gvec support for rotate by vector
No host backend support yet, but the interfaces for rotlv
and rotrv are in place.

Reviewed-by: Alex Bennée <alex.bennee@linaro.org>
Signed-off-by: Richard Henderson <richard.henderson@linaro.org>
---
v3: Drop the generic expansion from rot to shift; we can do better
for each backend, and then this code becomes unused.

Backports commit 5d0ceda902915e3f0e21c39d142c92c4e97c3ebb from qemu
2020-06-14 21:43:46 -04:00
Richard Henderson 5cce52a04b tcg: Implement gvec support for rotate by immediate
No host backend support yet, but the interfaces for rotli
are in place. Canonicalize immediate rotate to the left,
based on a survey of architectures, but provide both left
and right shift interfaces to the translators.

Backports commit b0f7e7444c03da17e41bf327c8aea590104a28ab from qemu
2020-06-14 21:26:58 -04:00
Peter Maydell bb0aa79847 target/arm: Convert Neon VADD, VSUB, VABD 3-reg-same insns to decodetree
Convert the Neon VADD, VSUB, VABD 3-reg-same insns to decodetree.
We already have gvec helpers for addition and subtraction, but must
add one for fabd.

Backports commit a26a352bb498662cd0c205cb433a352f86fac7d2 from qemu
2020-05-15 23:26:51 -04:00
Richard Henderson 451683ee79 target/arm: Vectorize SABA/UABA
Include 64-bit element size in preparation for SVE2.

Backports commit cfdb2c0c95ae9205b0dd7f0f5e970cdec50fef20 from qemu
2020-05-15 22:15:14 -04:00
Richard Henderson 98c79f9afc target/arm: Vectorize SABD/UABD
Include 64-bit element size in preparation for SVE2.

Backports commit 50c160d44eb059c7fc7f348ae2c3b0cb41437044 from qemu
2020-05-15 22:01:29 -04:00
Richard Henderson 3c4f226e00 target/arm: Create gen_gvec_{qrdmla,qrdmls}
Provide a functional interface for the vector expansion.
This fits better with the existing set of helpers that
we provide for other operations.

Backports commit 146aa66ce58b686b8037d0eb3921c1125942dbde from qemu
2020-05-15 21:43:22 -04:00
Richard Henderson 9dfc0479ff target/arm: Create gen_gvec_{uqadd, sqadd, uqsub, sqsub}
Provide a functional interface for the vector expansion.
This fits better with the existing set of helpers that
we provide for other operations.

Backports commit c7715b6b51a6f7a5412c5fcb40a4c8586105e597 from qemu
2020-05-15 21:25:06 -04:00
Richard Henderson 4abfe5156d target/arm: Create gen_gvec_{cmtst,ushl,sshl}
Provide a functional interface for the vector expansion.
This fits better with the existing set of helpers that
we provide for other operations.

Backports commit 8161b75357095fef54c76b1a6ed1e54d0e8655e0 from qemu
2020-05-15 21:15:49 -04:00
Richard Henderson 546db9089c target/arm: Create gen_gvec_{mla,mls}
Provide a functional interface for the vector expansion.
This fits better with the existing set of helpers that
we provide for other operations.

Backports commit 271063206a46062a45fc6bab8dabe45f0b88159d from qemu
2020-05-15 21:06:06 -04:00
Richard Henderson 340f97bf4c target/arm: Create gen_gvec_{ceq,clt,cle,cgt,cge}0
Provide a functional interface for the vector expansion.
This fits better with the existing set of helpers that
we provide for other operations.

Macro-ize the 5 nearly identical comparisons.

Backports commit 69d5e2bf8c3cefedbfa1c1670137e636dbd7faa5 from qemu
2020-05-15 20:57:33 -04:00
Richard Henderson 6190be3191 target/arm: Create gen_gvec_{sri,sli}
The functions eliminate duplication of the special cases for
this operation. They match up with the GVecGen2iFn typedef.

Add out-of-line helpers. We got away with only having inline
expanders because the neon vector size is only 16 bytes, and
we know that the inline expansion will always succeed.
When we reuse this for SVE, tcg-gvec-op may decide to use an
out-of-line helper due to longer vector lengths.

Backports commit 893ab0542aa385a287cbe46d5535c8b9e95ce699 from qemu
2020-05-15 20:39:28 -04:00
Richard Henderson 2609e6f319 target/arm: Create gen_gvec_{u,s}{rshr,rsra}
Create vectorized versions of handle_shri_with_rndacc
for shift+round and shift+round+accumulate. Add out-of-line
helpers in preparation for longer vector lengths from SVE.

Backports commit 6ccd48d4ea244c1c46a24dfa50bfb547f11422dd from qemu
2020-05-15 20:28:44 -04:00
Richard Henderson 5d7c46204d target/arm: Create gen_gvec_[us]sra
The functions eliminate duplication of the special cases for
this operation. They match up with the GVecGen2iFn typedef.

Add out-of-line helpers. We got away with only having inline
expanders because the neon vector size is only 16 bytes, and
we know that the inline expansion will always succeed.
When we reuse this for SVE, tcg-gvec-op may decide to use an
out-of-line helper due to longer vector lengths.

Backports commit 631e565450c483e0622eec3d8b61d7fa41d16bca from qemu
2020-05-15 20:10:32 -04:00
Richard Henderson e65806c356 tcg: Remove tcg_gen_gvec_dup{8,16,32,64}i
These interfaces are now unused.

Backports commit 398f21412aeec158338963e3f71c9313bc126a71 form qemu
2020-05-07 10:11:00 -04:00
Richard Henderson 07f622e57d tcg: Add tcg_gen_gvec_dup_imm
Add a version of tcg_gen_dup_* that takes both immediate and
a vector element size operand. This will replace the set of
tcg_gen_gvec_dup{8,16,32,64}i functions that encode the element
size within the function name.

Backports commit 44c94677febd15488f9190b11eaa4a08e8ac696b from qemu
2020-05-07 09:55:25 -04:00