Skip to content

Commit c8eb535

Browse files
eopXDeopXD
committed
[1/11][IR] Permit load/store/alloca for struct of the same scalable vector type
This patch-set aims to simplify the existing RVV segment load/store intrinsics to use a type that represents a tuple of vectors instead. To achieve this, first we need to relax the current limitation for an aggregate type to be a target of load/store/alloca when the aggregate type contains homogeneous scalable vector types. Then to adjust the prolog of an LLVM function during lowering to clang. Finally we re-define the RVV segment load/store intrinsics to use the tuple types. The pull request under the RVV intrinsic specification is riscv-non-isa/rvv-intrinsic-doc#198 --- This is the 1st patch of the patch-set. This patch is originated from D98169. This patch allows aggregate type (StructType) that contains homogeneous scalable vector types to be a target of load/store/alloca. The RFC of this patch was posted in LLVM Discourse. https://discourse.llvm.org/t/rfc-ir-permit-load-store-alloca-for-struct-of-the-same-scalable-vector-type/69527 The main changes in this patch are: Extend `StructLayout::StructSize` from `uint64_t` to `TypeSize` to accommodate an expression of scalable size. Allow `StructType:isSized` to also return true for homogeneous scalable vector types. Let `Type::isScalableTy` return true when `Type` is `StructType` and contains scalable vectors Extra description is added in the LLVM Language Reference Manual on the relaxation of this patch. Authored-by: Hsiangkai Wang <kai.wang@sifive.com> Co-Authored-by: eop Chen <eop.chen@sifive.com> Reviewed By: craig.topper, nikic Differential Revision: https://reviews.llvm.org/D146872
1 parent dae1754 commit c8eb535

23 files changed

+415
-73
lines changed

‎llvm/docs/LangRef.rst

Lines changed: 13 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -744,8 +744,14 @@ Variables and aliases can have a
744744

745745
:ref:`Scalable vectors <t_vector>` cannot be global variables or members of
746746
arrays because their size is unknown at compile time. They are allowed in
747-
structs to facilitate intrinsics returning multiple values. Structs containing
748-
scalable vectors cannot be used in loads, stores, allocas, or GEPs.
747+
structs to facilitate intrinsics returning multiple values. Generally, structs
748+
containing scalable vectors are not considered "sized" and cannot be used in
749+
loads, stores, allocas, or GEPs. The only exception to this rule is for structs
750+
that contain scalable vectors of the same type (e.g. ``{<vscale x 2 x i32>,
751+
<vscale x 2 x i32>}`` contains the same type while ``{<vscale x 2 x i32>,
752+
<vscale x 2 x i64>}`` doesn't). These kinds of structs (we may call them
753+
homogeneous scalable vector structs) are considered sized and can be used in
754+
loads, stores, allocas, but not GEPs.
749755

750756
Syntax::
751757

@@ -10287,6 +10293,11 @@ allocation on any convenient boundary compatible with the type.
1028710293

1028810294
'``type``' may be any sized type.
1028910295

10296+
Structs containing scalable vectors cannot be used in allocas unless all
10297+
fields are the same scalable vector type (e.g. ``{<vscale x 2 x i32>,
10298+
<vscale x 2 x i32>}`` contains the same type while ``{<vscale x 2 x i32>,
10299+
<vscale x 2 x i64>}`` doesn't).
10300+
1029010301
Semantics:
1029110302
""""""""""
1029210303

‎llvm/include/llvm/CodeGen/Analysis.h

Lines changed: 20 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -64,15 +64,33 @@ inline unsigned ComputeLinearIndex(Type *Ty,
6464
///
6565
void ComputeValueVTs(const TargetLowering &TLI, const DataLayout &DL, Type *Ty,
6666
SmallVectorImpl<EVT> &ValueVTs,
67-
SmallVectorImpl<uint64_t> *Offsets = nullptr,
67+
SmallVectorImpl<TypeSize> *Offsets,
68+
TypeSize StartingOffset);
69+
void ComputeValueVTs(const TargetLowering &TLI, const DataLayout &DL, Type *Ty,
70+
SmallVectorImpl<EVT> &ValueVTs,
71+
SmallVectorImpl<TypeSize> *Offsets = nullptr,
6872
uint64_t StartingOffset = 0);
73+
void ComputeValueVTs(const TargetLowering &TLI, const DataLayout &DL, Type *Ty,
74+
SmallVectorImpl<EVT> &ValueVTs,
75+
SmallVectorImpl<uint64_t> *FixedOffsets,
76+
uint64_t StartingOffset);
6977

7078
/// Variant of ComputeValueVTs that also produces the memory VTs.
7179
void ComputeValueVTs(const TargetLowering &TLI, const DataLayout &DL, Type *Ty,
7280
SmallVectorImpl<EVT> &ValueVTs,
7381
SmallVectorImpl<EVT> *MemVTs,
74-
SmallVectorImpl<uint64_t> *Offsets = nullptr,
82+
SmallVectorImpl<TypeSize> *Offsets,
83+
TypeSize StartingOffset);
84+
void ComputeValueVTs(const TargetLowering &TLI, const DataLayout &DL, Type *Ty,
85+
SmallVectorImpl<EVT> &ValueVTs,
86+
SmallVectorImpl<EVT> *MemVTs,
87+
SmallVectorImpl<TypeSize> *Offsets = nullptr,
7588
uint64_t StartingOffset = 0);
89+
void ComputeValueVTs(const TargetLowering &TLI, const DataLayout &DL, Type *Ty,
90+
SmallVectorImpl<EVT> &ValueVTs,
91+
SmallVectorImpl<EVT> *MemVTs,
92+
SmallVectorImpl<uint64_t> *FixedOffsets,
93+
uint64_t StartingOffset);
7694

7795
/// computeValueLLTs - Given an LLVM IR type, compute a sequence of
7896
/// LLTs that represent all the individual underlying

‎llvm/include/llvm/IR/DataLayout.h

Lines changed: 13 additions & 15 deletions
Original file line numberDiff line numberDiff line change
@@ -620,16 +620,16 @@ inline LLVMTargetDataRef wrap(const DataLayout *P) {
620620

621621
/// Used to lazily calculate structure layout information for a target machine,
622622
/// based on the DataLayout structure.
623-
class StructLayout final : public TrailingObjects<StructLayout, uint64_t> {
624-
uint64_t StructSize;
623+
class StructLayout final : public TrailingObjects<StructLayout, TypeSize> {
624+
TypeSize StructSize;
625625
Align StructAlignment;
626626
unsigned IsPadded : 1;
627627
unsigned NumElements : 31;
628628

629629
public:
630-
uint64_t getSizeInBytes() const { return StructSize; }
630+
TypeSize getSizeInBytes() const { return StructSize; }
631631

632-
uint64_t getSizeInBits() const { return 8 * StructSize; }
632+
TypeSize getSizeInBits() const { return 8 * StructSize; }
633633

634634
Align getAlignment() const { return StructAlignment; }
635635

@@ -639,23 +639,22 @@ class StructLayout final : public TrailingObjects<StructLayout, uint64_t> {
639639

640640
/// Given a valid byte offset into the structure, returns the structure
641641
/// index that contains it.
642-
unsigned getElementContainingOffset(uint64_t Offset) const;
642+
unsigned getElementContainingOffset(uint64_t FixedOffset) const;
643643

644-
MutableArrayRef<uint64_t> getMemberOffsets() {
645-
return llvm::MutableArrayRef(getTrailingObjects<uint64_t>(),
646-
NumElements);
644+
MutableArrayRef<TypeSize> getMemberOffsets() {
645+
return llvm::MutableArrayRef(getTrailingObjects<TypeSize>(), NumElements);
647646
}
648647

649-
ArrayRef<uint64_t> getMemberOffsets() const {
650-
return llvm::ArrayRef(getTrailingObjects<uint64_t>(), NumElements);
648+
ArrayRef<TypeSize> getMemberOffsets() const {
649+
return llvm::ArrayRef(getTrailingObjects<TypeSize>(), NumElements);
651650
}
652651

653-
uint64_t getElementOffset(unsigned Idx) const {
652+
TypeSize getElementOffset(unsigned Idx) const {
654653
assert(Idx < NumElements && "Invalid element idx!");
655654
return getMemberOffsets()[Idx];
656655
}
657656

658-
uint64_t getElementOffsetInBits(unsigned Idx) const {
657+
TypeSize getElementOffsetInBits(unsigned Idx) const {
659658
return getElementOffset(Idx) * 8;
660659
}
661660

@@ -664,7 +663,7 @@ class StructLayout final : public TrailingObjects<StructLayout, uint64_t> {
664663

665664
StructLayout(StructType *ST, const DataLayout &DL);
666665

667-
size_t numTrailingObjects(OverloadToken<uint64_t>) const {
666+
size_t numTrailingObjects(OverloadToken<TypeSize>) const {
668667
return NumElements;
669668
}
670669
};
@@ -685,8 +684,7 @@ inline TypeSize DataLayout::getTypeSizeInBits(Type *Ty) const {
685684
}
686685
case Type::StructTyID:
687686
// Get the layout annotation... which is lazily created on demand.
688-
return TypeSize::Fixed(
689-
getStructLayout(cast<StructType>(Ty))->getSizeInBits());
687+
return getStructLayout(cast<StructType>(Ty))->getSizeInBits();
690688
case Type::IntegerTyID:
691689
return TypeSize::Fixed(Ty->getIntegerBitWidth());
692690
case Type::HalfTyID:

‎llvm/include/llvm/IR/DerivedTypes.h

Lines changed: 13 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -218,7 +218,9 @@ class StructType : public Type {
218218
SCDB_HasBody = 1,
219219
SCDB_Packed = 2,
220220
SCDB_IsLiteral = 4,
221-
SCDB_IsSized = 8
221+
SCDB_IsSized = 8,
222+
SCDB_ContainsScalableVector = 16,
223+
SCDB_NotContainsScalableVector = 32
222224
};
223225

224226
/// For a named struct that actually has a name, this is a pointer to the
@@ -284,7 +286,16 @@ class StructType : public Type {
284286
bool isSized(SmallPtrSetImpl<Type *> *Visited = nullptr) const;
285287

286288
/// Returns true if this struct contains a scalable vector.
287-
bool containsScalableVectorType() const;
289+
bool
290+
containsScalableVectorType(SmallPtrSetImpl<Type *> *Visited = nullptr) const;
291+
292+
/// Returns true if this struct contains homogeneous scalable vector types.
293+
/// Note that the definition of homogeneous scalable vector type is not
294+
/// recursive here. That means the following structure will return false
295+
/// when calling this function.
296+
/// {{<vscale x 2 x i32>, <vscale x 4 x i64>},
297+
/// {<vscale x 2 x i32>, <vscale x 4 x i64>}}
298+
bool containsHomogeneousScalableVectorTypes() const;
288299

289300
/// Return true if this is a named struct that has a non-empty name.
290301
bool hasName() const { return SymbolTableEntry != nullptr; }

‎llvm/include/llvm/IR/Type.h

Lines changed: 1 addition & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -211,9 +211,7 @@ class Type {
211211

212212
/// Return true if this is a scalable vector type or a target extension type
213213
/// with a scalable layout.
214-
bool isScalableTy() const {
215-
return getTypeID() == ScalableVectorTyID || isScalableTargetExtTy();
216-
}
214+
bool isScalableTy() const;
217215

218216
/// Return true if this is a FP type or a vector of FP.
219217
bool isFPOrFPVectorTy() const { return getScalarType()->isFloatingPointTy(); }

‎llvm/lib/Analysis/ScalarEvolution.cpp

Lines changed: 4 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -4345,8 +4345,10 @@ const SCEV *ScalarEvolution::getOffsetOfExpr(Type *IntTy,
43454345
// We can bypass creating a target-independent constant expression and then
43464346
// folding it back into a ConstantInt. This is just a compile-time
43474347
// optimization.
4348-
return getConstant(
4349-
IntTy, getDataLayout().getStructLayout(STy)->getElementOffset(FieldNo));
4348+
const StructLayout *SL = getDataLayout().getStructLayout(STy);
4349+
assert(!SL->getSizeInBits().isScalable() &&
4350+
"Cannot get offset for structure containing scalable vector types");
4351+
return getConstant(IntTy, SL->getElementOffset(FieldNo));
43504352
}
43514353

43524354
const SCEV *ScalarEvolution::getUnknown(Value *V) {

‎llvm/lib/AsmParser/LLParser.cpp

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -7990,6 +7990,11 @@ int LLParser::parseGetElementPtr(Instruction *&Inst, PerFunctionState &PFS) {
79907990
if (!Indices.empty() && !Ty->isSized(&Visited))
79917991
return error(Loc, "base element of getelementptr must be sized");
79927992

7993+
auto *STy = dyn_cast<StructType>(Ty);
7994+
if (STy && STy->containsScalableVectorType())
7995+
return error(Loc, "getelementptr cannot target structure that contains "
7996+
"scalable vector type");
7997+
79937998
if (!GetElementPtrInst::getIndexedType(Ty, Indices))
79947999
return error(Loc, "invalid getelementptr indices");
79958000
Inst = GetElementPtrInst::Create(Ty, Ptr, Indices);

‎llvm/lib/CodeGen/Analysis.cpp

Lines changed: 57 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -79,8 +79,8 @@ unsigned llvm::ComputeLinearIndex(Type *Ty,
7979
void llvm::ComputeValueVTs(const TargetLowering &TLI, const DataLayout &DL,
8080
Type *Ty, SmallVectorImpl<EVT> &ValueVTs,
8181
SmallVectorImpl<EVT> *MemVTs,
82-
SmallVectorImpl<uint64_t> *Offsets,
83-
uint64_t StartingOffset) {
82+
SmallVectorImpl<TypeSize> *Offsets,
83+
TypeSize StartingOffset) {
8484
// Given a struct type, recursively traverse the elements.
8585
if (StructType *STy = dyn_cast<StructType>(Ty)) {
8686
// If the Offsets aren't needed, don't query the struct layout. This allows
@@ -92,7 +92,8 @@ void llvm::ComputeValueVTs(const TargetLowering &TLI, const DataLayout &DL,
9292
EE = STy->element_end();
9393
EI != EE; ++EI) {
9494
// Don't compute the element offset if we didn't get a StructLayout above.
95-
uint64_t EltOffset = SL ? SL->getElementOffset(EI - EB) : 0;
95+
TypeSize EltOffset = SL ? SL->getElementOffset(EI - EB)
96+
: TypeSize::get(0, StartingOffset.isScalable());
9697
ComputeValueVTs(TLI, DL, *EI, ValueVTs, MemVTs, Offsets,
9798
StartingOffset + EltOffset);
9899
}
@@ -101,7 +102,7 @@ void llvm::ComputeValueVTs(const TargetLowering &TLI, const DataLayout &DL,
101102
// Given an array type, recursively traverse the elements.
102103
if (ArrayType *ATy = dyn_cast<ArrayType>(Ty)) {
103104
Type *EltTy = ATy->getElementType();
104-
uint64_t EltSize = DL.getTypeAllocSize(EltTy).getFixedValue();
105+
TypeSize EltSize = DL.getTypeAllocSize(EltTy);
105106
for (unsigned i = 0, e = ATy->getNumElements(); i != e; ++i)
106107
ComputeValueVTs(TLI, DL, EltTy, ValueVTs, MemVTs, Offsets,
107108
StartingOffset + i * EltSize);
@@ -120,12 +121,62 @@ void llvm::ComputeValueVTs(const TargetLowering &TLI, const DataLayout &DL,
120121

121122
void llvm::ComputeValueVTs(const TargetLowering &TLI, const DataLayout &DL,
122123
Type *Ty, SmallVectorImpl<EVT> &ValueVTs,
123-
SmallVectorImpl<uint64_t> *Offsets,
124-
uint64_t StartingOffset) {
124+
SmallVectorImpl<TypeSize> *Offsets,
125+
TypeSize StartingOffset) {
125126
return ComputeValueVTs(TLI, DL, Ty, ValueVTs, /*MemVTs=*/nullptr, Offsets,
126127
StartingOffset);
127128
}
128129

130+
void llvm::ComputeValueVTs(const TargetLowering &TLI, const DataLayout &DL,
131+
Type *Ty, SmallVectorImpl<EVT> &ValueVTs,
132+
SmallVectorImpl<TypeSize> *Offsets,
133+
uint64_t StartingOffset) {
134+
TypeSize Offset = TypeSize::get(StartingOffset, Ty->isScalableTy());
135+
return ComputeValueVTs(TLI, DL, Ty, ValueVTs, Offsets, Offset);
136+
}
137+
138+
void llvm::ComputeValueVTs(const TargetLowering &TLI, const DataLayout &DL,
139+
Type *Ty, SmallVectorImpl<EVT> &ValueVTs,
140+
SmallVectorImpl<uint64_t> *FixedOffsets,
141+
uint64_t StartingOffset) {
142+
TypeSize Offset = TypeSize::get(StartingOffset, Ty->isScalableTy());
143+
SmallVector<TypeSize, 4> Offsets;
144+
if (FixedOffsets)
145+
ComputeValueVTs(TLI, DL, Ty, ValueVTs, &Offsets, Offset);
146+
else
147+
ComputeValueVTs(TLI, DL, Ty, ValueVTs, nullptr, Offset);
148+
149+
if (FixedOffsets)
150+
for (TypeSize Offset : Offsets)
151+
FixedOffsets->push_back(Offset.getKnownMinValue());
152+
}
153+
154+
void llvm::ComputeValueVTs(const TargetLowering &TLI, const DataLayout &DL,
155+
Type *Ty, SmallVectorImpl<EVT> &ValueVTs,
156+
SmallVectorImpl<EVT> *MemVTs,
157+
SmallVectorImpl<TypeSize> *Offsets,
158+
uint64_t StartingOffset) {
159+
TypeSize Offset = TypeSize::get(StartingOffset, Ty->isScalableTy());
160+
return ComputeValueVTs(TLI, DL, Ty, ValueVTs, MemVTs, Offsets, Offset);
161+
}
162+
163+
void llvm::ComputeValueVTs(const TargetLowering &TLI, const DataLayout &DL,
164+
Type *Ty, SmallVectorImpl<EVT> &ValueVTs,
165+
SmallVectorImpl<EVT> *MemVTs,
166+
SmallVectorImpl<uint64_t> *FixedOffsets,
167+
uint64_t StartingOffset) {
168+
TypeSize Offset = TypeSize::get(StartingOffset, Ty->isScalableTy());
169+
SmallVector<TypeSize, 4> Offsets;
170+
if (FixedOffsets)
171+
ComputeValueVTs(TLI, DL, Ty, ValueVTs, MemVTs, &Offsets, Offset);
172+
else
173+
ComputeValueVTs(TLI, DL, Ty, ValueVTs, MemVTs, nullptr, Offset);
174+
175+
if (FixedOffsets)
176+
for (TypeSize Offset : Offsets)
177+
FixedOffsets->push_back(Offset.getKnownMinValue());
178+
}
179+
129180
void llvm::computeValueLLTs(const DataLayout &DL, Type &Ty,
130181
SmallVectorImpl<LLT> &ValueTys,
131182
SmallVectorImpl<uint64_t> *Offsets,

‎llvm/lib/CodeGen/SelectionDAG/FunctionLoweringInfo.cpp

Lines changed: 4 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -152,9 +152,10 @@ void FunctionLoweringInfo::set(const Function &fn, MachineFunction &mf,
152152
false, AI);
153153
}
154154

155-
// Scalable vectors may need a special StackID to distinguish
156-
// them from other (fixed size) stack objects.
157-
if (isa<ScalableVectorType>(Ty))
155+
// Scalable vectors and structures that contain scalable vectors may
156+
// need a special StackID to distinguish them from other (fixed size)
157+
// stack objects.
158+
if (Ty->isScalableTy())
158159
MF->getFrameInfo().setStackID(FrameIndex,
159160
TFI->getStackIDForScalableVectors());
160161

‎llvm/lib/CodeGen/SelectionDAG/SelectionDAGBuilder.cpp

Lines changed: 6 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -2025,7 +2025,7 @@ void SelectionDAGBuilder::visitRet(const ReturnInst &I) {
20252025
SmallVector<EVT, 4> ValueVTs, MemVTs;
20262026
SmallVector<uint64_t, 4> Offsets;
20272027
ComputeValueVTs(TLI, DL, I.getOperand(0)->getType(), ValueVTs, &MemVTs,
2028-
&Offsets);
2028+
&Offsets, 0);
20292029
unsigned NumValues = ValueVTs.size();
20302030

20312031
SmallVector<SDValue, 4> Chains(NumValues);
@@ -4161,7 +4161,7 @@ void SelectionDAGBuilder::visitLoad(const LoadInst &I) {
41614161
Type *Ty = I.getType();
41624162
SmallVector<EVT, 4> ValueVTs, MemVTs;
41634163
SmallVector<uint64_t, 4> Offsets;
4164-
ComputeValueVTs(TLI, DAG.getDataLayout(), Ty, ValueVTs, &MemVTs, &Offsets);
4164+
ComputeValueVTs(TLI, DAG.getDataLayout(), Ty, ValueVTs, &MemVTs, &Offsets, 0);
41654165
unsigned NumValues = ValueVTs.size();
41664166
if (NumValues == 0)
41674167
return;
@@ -4260,7 +4260,7 @@ void SelectionDAGBuilder::visitStoreToSwiftError(const StoreInst &I) {
42604260
SmallVector<uint64_t, 4> Offsets;
42614261
const Value *SrcV = I.getOperand(0);
42624262
ComputeValueVTs(DAG.getTargetLoweringInfo(), DAG.getDataLayout(),
4263-
SrcV->getType(), ValueVTs, &Offsets);
4263+
SrcV->getType(), ValueVTs, &Offsets, 0);
42644264
assert(ValueVTs.size() == 1 && Offsets[0] == 0 &&
42654265
"expect a single EVT for swifterror");
42664266

@@ -4296,7 +4296,7 @@ void SelectionDAGBuilder::visitLoadFromSwiftError(const LoadInst &I) {
42964296
SmallVector<EVT, 4> ValueVTs;
42974297
SmallVector<uint64_t, 4> Offsets;
42984298
ComputeValueVTs(DAG.getTargetLoweringInfo(), DAG.getDataLayout(), Ty,
4299-
ValueVTs, &Offsets);
4299+
ValueVTs, &Offsets, 0);
43004300
assert(ValueVTs.size() == 1 && Offsets[0] == 0 &&
43014301
"expect a single EVT for swifterror");
43024302

@@ -4333,7 +4333,7 @@ void SelectionDAGBuilder::visitStore(const StoreInst &I) {
43334333
SmallVector<EVT, 4> ValueVTs, MemVTs;
43344334
SmallVector<uint64_t, 4> Offsets;
43354335
ComputeValueVTs(DAG.getTargetLoweringInfo(), DAG.getDataLayout(),
4336-
SrcV->getType(), ValueVTs, &MemVTs, &Offsets);
4336+
SrcV->getType(), ValueVTs, &MemVTs, &Offsets, 0);
43374337
unsigned NumValues = ValueVTs.size();
43384338
if (NumValues == 0)
43394339
return;
@@ -9903,7 +9903,7 @@ TargetLowering::LowerCallTo(TargetLowering::CallLoweringInfo &CLI) const {
99039903
SmallVector<EVT, 4> RetTys;
99049904
SmallVector<uint64_t, 4> Offsets;
99059905
auto &DL = CLI.DAG.getDataLayout();
9906-
ComputeValueVTs(*this, DL, CLI.RetTy, RetTys, &Offsets);
9906+
ComputeValueVTs(*this, DL, CLI.RetTy, RetTys, &Offsets, 0);
99079907

99089908
if (CLI.IsPostTypeLegalization) {
99099909
// If we are lowering a libcall after legalization, split the return type.

0 commit comments

Comments
 (0)