[VPlan] Construct initial once and pass clones to tryToBuildVPlan (NFC). #141363

fhahn · 2025-05-24T18:34:15Z

Update to only build an initial, plain-CFG VPlan once, and then transform & optimize clones.

This requires changes to ::clone() for VPInstruction and VPWidenPHIRecipe to allow for proper cloning of the recipes in the initial VPlan.

Update to only build an initial, plain-CFG VPlan once, and then transform & optimize clones. This requires changes to ::clone() for VPInstruction and VPWidenPHIRecipe to allow for proper cloning of the recipes in the initial VPlan.

llvmbot · 2025-05-24T18:34:49Z

@llvm/pr-subscribers-vectorizers

Author: Florian Hahn (fhahn)

Changes

Update to only build an initial, plain-CFG VPlan once, and then transform & optimize clones.

This requires changes to ::clone() for VPInstruction and VPWidenPHIRecipe to allow for proper cloning of the recipes in the initial VPlan.

Full diff: https://github.com/llvm/llvm-project/pull/141363.diff

3 Files Affected:

(modified) llvm/lib/Transforms/Vectorize/LoopVectorizationPlanner.h (+2-1)
(modified) llvm/lib/Transforms/Vectorize/LoopVectorize.cpp (+5-5)
(modified) llvm/lib/Transforms/Vectorize/VPlan.h (+7-1)

diff --git a/llvm/lib/Transforms/Vectorize/LoopVectorizationPlanner.h b/llvm/lib/Transforms/Vectorize/LoopVectorizationPlanner.h
index bae53c600c18c..c772c74113d97 100644
--- a/llvm/lib/Transforms/Vectorize/LoopVectorizationPlanner.h
+++ b/llvm/lib/Transforms/Vectorize/LoopVectorizationPlanner.h
@@ -524,7 +524,8 @@ class LoopVectorizationPlanner {
   /// returned VPlan is valid for. If no VPlan can be built for the input range,
   /// set the largest included VF to the maximum VF for which no plan could be
   /// built.
-  VPlanPtr tryToBuildVPlanWithVPRecipes(VFRange &Range, LoopVersioning *LVer);
+  VPlanPtr tryToBuildVPlanWithVPRecipes(VPlanPtr InitialPlan, VFRange &Range,
+                                        LoopVersioning *LVer);
 
   /// Build VPlans for power-of-2 VF's between \p MinVF and \p MaxVF inclusive,
   /// according to the information gathered by Legal when it checked if it is
diff --git a/llvm/lib/Transforms/Vectorize/LoopVectorize.cpp b/llvm/lib/Transforms/Vectorize/LoopVectorize.cpp
index 8636550d4f644..1232538e68dd0 100644
--- a/llvm/lib/Transforms/Vectorize/LoopVectorize.cpp
+++ b/llvm/lib/Transforms/Vectorize/LoopVectorize.cpp
@@ -8715,11 +8715,13 @@ void LoopVectorizationPlanner::buildVPlansWithVPRecipes(ElementCount MinVF,
     // overlap across all iterations.
     LVer.prepareNoAliasMetadata();
   }
+  auto VPlan0 = VPlanTransforms::buildPlainCFG(OrigLoop, *LI);
 
   auto MaxVFTimes2 = MaxVF * 2;
   for (ElementCount VF = MinVF; ElementCount::isKnownLT(VF, MaxVFTimes2);) {
     VFRange SubRange = {VF, MaxVFTimes2};
-    if (auto Plan = tryToBuildVPlanWithVPRecipes(SubRange, &LVer)) {
+    if (auto Plan = tryToBuildVPlanWithVPRecipes(
+            std::unique_ptr<VPlan>(VPlan0->duplicate()), SubRange, &LVer)) {
       bool HasScalarVF = Plan->hasScalarVFOnly();
       // Now optimize the initial VPlan.
       if (!HasScalarVF)
@@ -8980,9 +8982,8 @@ static void addExitUsersForFirstOrderRecurrences(
   }
 }
 
-VPlanPtr
-LoopVectorizationPlanner::tryToBuildVPlanWithVPRecipes(VFRange &Range,
-                                                       LoopVersioning *LVer) {
+VPlanPtr LoopVectorizationPlanner::tryToBuildVPlanWithVPRecipes(
+    VPlanPtr Plan, VFRange &Range, LoopVersioning *LVer) {
 
   using namespace llvm::VPlanPatternMatch;
   SmallPtrSet<const InterleaveGroup<Instruction> *, 1> InterleaveGroups;
@@ -9004,7 +9005,6 @@ LoopVectorizationPlanner::tryToBuildVPlanWithVPRecipes(VFRange &Range,
             return !CM.requiresScalarEpilogue(VF.isVector());
           },
           Range);
-  auto Plan = VPlanTransforms::buildPlainCFG(OrigLoop, *LI);
   VPlanTransforms::prepareForVectorization(
       *Plan, Legal->getWidestInductionType(), PSE, RequiresScalarEpilogueCheck,
       CM.foldTailByMasking(), OrigLoop,
diff --git a/llvm/lib/Transforms/Vectorize/VPlan.h b/llvm/lib/Transforms/Vectorize/VPlan.h
index c4e66cd89e69c..940df596e5cec 100644
--- a/llvm/lib/Transforms/Vectorize/VPlan.h
+++ b/llvm/lib/Transforms/Vectorize/VPlan.h
@@ -1004,6 +1004,8 @@ class VPInstruction : public VPRecipeWithIRFlags,
   VPInstruction *clone() override {
     SmallVector<VPValue *, 2> Operands(operands());
     auto *New = new VPInstruction(Opcode, Operands, getDebugLoc(), Name);
+    if (getUnderlyingValue())
+      New->setUnderlyingValue(getUnderlyingInstr());
     New->transferFlags(*this);
     return New;
   }
@@ -2129,7 +2131,11 @@ class VPWidenPHIRecipe : public VPSingleDefRecipe, public VPPhiAccessors {
   }
 
   VPWidenPHIRecipe *clone() override {
-    llvm_unreachable("cloning not implemented yet");
+    auto *C = new VPWidenPHIRecipe(cast<PHINode>(getUnderlyingValue()),
+                                   getOperand(0), getDebugLoc(), Name);
+    for (VPValue *Op : make_range(std::next(op_begin()), op_end()))
+      C->addOperand(Op);
+    return C;
   }
 
   ~VPWidenPHIRecipe() override = default;

llvmbot · 2025-05-24T18:34:49Z

@llvm/pr-subscribers-llvm-transforms

Author: Florian Hahn (fhahn)

Changes

Update to only build an initial, plain-CFG VPlan once, and then transform & optimize clones.

This requires changes to ::clone() for VPInstruction and VPWidenPHIRecipe to allow for proper cloning of the recipes in the initial VPlan.

Full diff: https://github.com/llvm/llvm-project/pull/141363.diff

3 Files Affected:

(modified) llvm/lib/Transforms/Vectorize/LoopVectorizationPlanner.h (+2-1)
(modified) llvm/lib/Transforms/Vectorize/LoopVectorize.cpp (+5-5)
(modified) llvm/lib/Transforms/Vectorize/VPlan.h (+7-1)

diff --git a/llvm/lib/Transforms/Vectorize/LoopVectorizationPlanner.h b/llvm/lib/Transforms/Vectorize/LoopVectorizationPlanner.h
index bae53c600c18c..c772c74113d97 100644
--- a/llvm/lib/Transforms/Vectorize/LoopVectorizationPlanner.h
+++ b/llvm/lib/Transforms/Vectorize/LoopVectorizationPlanner.h
@@ -524,7 +524,8 @@ class LoopVectorizationPlanner {
   /// returned VPlan is valid for. If no VPlan can be built for the input range,
   /// set the largest included VF to the maximum VF for which no plan could be
   /// built.
-  VPlanPtr tryToBuildVPlanWithVPRecipes(VFRange &Range, LoopVersioning *LVer);
+  VPlanPtr tryToBuildVPlanWithVPRecipes(VPlanPtr InitialPlan, VFRange &Range,
+                                        LoopVersioning *LVer);
 
   /// Build VPlans for power-of-2 VF's between \p MinVF and \p MaxVF inclusive,
   /// according to the information gathered by Legal when it checked if it is
diff --git a/llvm/lib/Transforms/Vectorize/LoopVectorize.cpp b/llvm/lib/Transforms/Vectorize/LoopVectorize.cpp
index 8636550d4f644..1232538e68dd0 100644
--- a/llvm/lib/Transforms/Vectorize/LoopVectorize.cpp
+++ b/llvm/lib/Transforms/Vectorize/LoopVectorize.cpp
@@ -8715,11 +8715,13 @@ void LoopVectorizationPlanner::buildVPlansWithVPRecipes(ElementCount MinVF,
     // overlap across all iterations.
     LVer.prepareNoAliasMetadata();
   }
+  auto VPlan0 = VPlanTransforms::buildPlainCFG(OrigLoop, *LI);
 
   auto MaxVFTimes2 = MaxVF * 2;
   for (ElementCount VF = MinVF; ElementCount::isKnownLT(VF, MaxVFTimes2);) {
     VFRange SubRange = {VF, MaxVFTimes2};
-    if (auto Plan = tryToBuildVPlanWithVPRecipes(SubRange, &LVer)) {
+    if (auto Plan = tryToBuildVPlanWithVPRecipes(
+            std::unique_ptr<VPlan>(VPlan0->duplicate()), SubRange, &LVer)) {
       bool HasScalarVF = Plan->hasScalarVFOnly();
       // Now optimize the initial VPlan.
       if (!HasScalarVF)
@@ -8980,9 +8982,8 @@ static void addExitUsersForFirstOrderRecurrences(
   }
 }
 
-VPlanPtr
-LoopVectorizationPlanner::tryToBuildVPlanWithVPRecipes(VFRange &Range,
-                                                       LoopVersioning *LVer) {
+VPlanPtr LoopVectorizationPlanner::tryToBuildVPlanWithVPRecipes(
+    VPlanPtr Plan, VFRange &Range, LoopVersioning *LVer) {
 
   using namespace llvm::VPlanPatternMatch;
   SmallPtrSet<const InterleaveGroup<Instruction> *, 1> InterleaveGroups;
@@ -9004,7 +9005,6 @@ LoopVectorizationPlanner::tryToBuildVPlanWithVPRecipes(VFRange &Range,
             return !CM.requiresScalarEpilogue(VF.isVector());
           },
           Range);
-  auto Plan = VPlanTransforms::buildPlainCFG(OrigLoop, *LI);
   VPlanTransforms::prepareForVectorization(
       *Plan, Legal->getWidestInductionType(), PSE, RequiresScalarEpilogueCheck,
       CM.foldTailByMasking(), OrigLoop,
diff --git a/llvm/lib/Transforms/Vectorize/VPlan.h b/llvm/lib/Transforms/Vectorize/VPlan.h
index c4e66cd89e69c..940df596e5cec 100644
--- a/llvm/lib/Transforms/Vectorize/VPlan.h
+++ b/llvm/lib/Transforms/Vectorize/VPlan.h
@@ -1004,6 +1004,8 @@ class VPInstruction : public VPRecipeWithIRFlags,
   VPInstruction *clone() override {
     SmallVector<VPValue *, 2> Operands(operands());
     auto *New = new VPInstruction(Opcode, Operands, getDebugLoc(), Name);
+    if (getUnderlyingValue())
+      New->setUnderlyingValue(getUnderlyingInstr());
     New->transferFlags(*this);
     return New;
   }
@@ -2129,7 +2131,11 @@ class VPWidenPHIRecipe : public VPSingleDefRecipe, public VPPhiAccessors {
   }
 
   VPWidenPHIRecipe *clone() override {
-    llvm_unreachable("cloning not implemented yet");
+    auto *C = new VPWidenPHIRecipe(cast<PHINode>(getUnderlyingValue()),
+                                   getOperand(0), getDebugLoc(), Name);
+    for (VPValue *Op : make_range(std::next(op_begin()), op_end()))
+      C->addOperand(Op);
+    return C;
   }
 
   ~VPWidenPHIRecipe() override = default;

ayalz

Nice hoisting of buildPlainCFG() out of tryToBuildVPlanWithVPRecipes(), ship it!

ayalz · 2025-05-25T21:01:56Z

llvm/lib/Transforms/Vectorize/LoopVectorizationPlanner.h

@@ -524,7 +524,8 @@ class LoopVectorizationPlanner {
  /// returned VPlan is valid for. If no VPlan can be built for the input range,
  /// set the largest included VF to the maximum VF for which no plan could be
  /// built.


Worth updating the documentation, say by adding something like:

Suggested change

/// built.

/// built. Each VPlan is built starting from a copy of \p InitialPlan, which is a plain CFG VPlan wrapping the original scalar loop.

?

Added thanks

ayalz · 2025-05-25T21:07:45Z

llvm/lib/Transforms/Vectorize/LoopVectorize.cpp

@@ -8715,11 +8715,13 @@ void LoopVectorizationPlanner::buildVPlansWithVPRecipes(ElementCount MinVF,
    // overlap across all iterations.
    LVer.prepareNoAliasMetadata();
  }
+  auto VPlan0 = VPlanTransforms::buildPlainCFG(OrigLoop, *LI);


Suggested change

auto VPlan0 = VPlanTransforms::buildPlainCFG(OrigLoop, *LI);

auto VPlan0 = VPlanTransforms::buildPlainCFG(OrigLoop, *LI);

moved down, thanks!

fhahn

Next follow-up would be checking what is needed to pull out also early region creation

…ldVPlan (NFC). (#141363) Update to only build an initial, plain-CFG VPlan once, and then transform & optimize clones. This requires changes to ::clone() for VPInstruction and VPWidenPHIRecipe to allow for proper cloning of the recipes in the initial VPlan. PR: llvm/llvm-project#141363

nikic · 2025-05-26T14:21:57Z

FYI this has some overhead, though not a lot: https://llvm-compile-time-tracker.com/compare.php?from=435d8b12efc1447b3c9f95ace2c57dae22ea486d&to=567b3172da2d52f5df70a37f3de06b7000b25968&stat=instructions:u I assume that's to be expected given what the change does.

…Plan (NFC). (llvm#141363)" This reverts commit 567b317.

fhahn · 2025-05-27T20:21:49Z

Thanks for the heads-up, it is a bit surprising. There was at least one case where we now are doing unnecessary work which should be fixed by ad58ea3.

This recovers some of the impact, but not yet all of it: http://llvm-compile-time-tracker.com/compare.php?from=3033f202f6707937cd28c2473479db134993f96f&to=1a0b9e5834f7fd4abf058864e656f8e26b7a26ff&stat=instructions:u

Will check if there's anything else to improve.

fhahn requested review from rengolin, ayalz and aniragil May 24, 2025 18:34

llvmbot added vectorizers llvm:transforms labels May 24, 2025

ayalz approved these changes May 25, 2025

View reviewed changes

fhahn commented May 25, 2025

View reviewed changes

fhahn added 2 commits May 26, 2025 12:38

Merge remote-tracking branch 'origin/main' into vplan-scalar-clone

7fd41e2

!fixup address latest comments, thanks

8253640

fhahn merged commit 567b317 into llvm:main May 26, 2025
11 checks passed

fhahn deleted the vplan-scalar-clone branch May 26, 2025 12:42

fhahn added a commit to fhahn/llvm-project that referenced this pull request May 26, 2025

Revert "[VPlan] Construct initial once and pass clones to tryToBuildV…

e87026b

…Plan (NFC). (llvm#141363)" This reverts commit 567b317.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[VPlan] Construct initial once and pass clones to tryToBuildVPlan (NFC). #141363

[VPlan] Construct initial once and pass clones to tryToBuildVPlan (NFC). #141363

Uh oh!

fhahn commented May 24, 2025

Uh oh!

llvmbot commented May 24, 2025

Uh oh!

llvmbot commented May 24, 2025

Uh oh!

ayalz left a comment

Uh oh!

ayalz May 25, 2025

Uh oh!

fhahn May 26, 2025

Uh oh!

ayalz May 25, 2025

Uh oh!

fhahn May 26, 2025

Uh oh!

fhahn left a comment

Uh oh!

Uh oh!

nikic commented May 26, 2025

Uh oh!

fhahn commented May 27, 2025

Uh oh!

Uh oh!

	/// built.
	/// built. Each VPlan is built starting from a copy of \p InitialPlan, which is a plain CFG VPlan wrapping the original scalar loop.

	auto VPlan0 = VPlanTransforms::buildPlainCFG(OrigLoop, *LI);

	auto VPlan0 = VPlanTransforms::buildPlainCFG(OrigLoop, *LI);

[VPlan] Construct initial once and pass clones to tryToBuildVPlan (NFC). #141363

[VPlan] Construct initial once and pass clones to tryToBuildVPlan (NFC). #141363

Uh oh!

Conversation

fhahn commented May 24, 2025

Uh oh!

llvmbot commented May 24, 2025

Uh oh!

llvmbot commented May 24, 2025

Uh oh!

ayalz left a comment

Choose a reason for hiding this comment

Uh oh!

ayalz May 25, 2025

Choose a reason for hiding this comment

Uh oh!

fhahn May 26, 2025

Choose a reason for hiding this comment

Uh oh!

ayalz May 25, 2025

Choose a reason for hiding this comment

Uh oh!

fhahn May 26, 2025

Choose a reason for hiding this comment

Uh oh!

fhahn left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

nikic commented May 26, 2025

Uh oh!

fhahn commented May 27, 2025

Uh oh!

Uh oh!