[Symfony 5.0 Forms]Surprising limitation: ModelTransformer cannot modify the data structure for a CollectionType

[spackmat]

Symfony version(s) affected: 5.0.7

Description
I have a CollectionType form field collecting a custom Type. The data structure from the model doesn't match the way the form should work, so I use a ModelTransformer to transform the data, reducing the count of items. This works fine, I can see that the field has the correct data structure in its "Normalized Format". But the collection now still has the count of items from the model data, while the transformed data is merged into that, leading to several empty items. I expected the collection to work (only) with the "Normalized Format" but in fact I get the count of items from the "Model Format" and the "Normalized Format" merged into them.

How to reproduce
Have a CollectionType form field and add a ModelTransformer that reduces the count of items from n to m. Then observe the form to render n items, with the first m having the transformed data and the other having null.

Possible Solution
I could switch the whole Form to use a DTO and transform the data there (a good idea anyway), but at first I want to have an opinion if this is a bug in the Form component.

Additional context
In my case, I see this data in the "Model Format" section of the profiler:

[▼
  [▼
    "month" => "01"
    "value" => 202001
    "fromYear" => "2020"
  ]
  [▼
    "month" => "02"
    "value" => 202002
    "fromYear" => "2020"
  ]
  [▼
    "month" => "03"
    "value" => 202003
    "fromYear" => "2020"
  ]
  [▼
    "month" => "04"
    "value" => 202004
    "fromYear" => "2020"
  ]
  [▼
    "month" => "05"
    "value" => 202005
    "fromYear" => "2020"
  ]
  [▼
    "month" => "06"
    "value" => 202006
    "fromYear" => "2020"
  ]
  [▼
    "month" => "07"
    "value" => 202007
    "fromYear" => "2020"
  ]
  [▼
    "month" => "08"
    "value" => 202008
    "fromYear" => "2020"
  ]
  [▼
    "month" => "09"
    "value" => 202009
    "fromYear" => "2020"
  ]
  [▼
    "month" => "10"
    "value" => 202010
    "fromYear" => "2020"
  ]
  [▼
    "month" => "11"
    "value" => 202011
    "fromYear" => "2020"
  ]
  [▼
    "month" => "12"
    "value" => 201912
    "fromYear" => "2019"
  ]
]

And this (expected) data in "Normalized Data" (that is what my transformer should do, as the custom type expects data for those 13 fields):

[▼
  [▼
    "fromYear" => "2019"
    "value01" => null
    "value02" => null
    "value04" => null
    "value05" => null
    "value06" => null
    "value07" => null
    "value08" => null
    "value09" => null
    "value10" => null
    "value11" => null
    "value12" => 201912
  ]
  [▼
    "fromYear" => "2020"
    "value01" => 202001
    "value02" => 202002
    "value03" => 202003
    "value04" => 202004
    "value05" => 202005
    "value06" => 202006
    "value07" => 202007
    "value08" => 202008
    "value09" => 202009
    "value10" => 202010
    "value11" => 202011
    "value12" => null
  ]
]

But the collection has 12 children instead of the expected 2 inclined by the Normalized Format:

Child 0 Normalized Format:

[▼
  "fromYear" => "2019"
  "value01" => null
  "value02" => null
  "value04" => null
  "value05" => null
  "value06" => null
  "value07" => null
  "value08" => null
  "value09" => null
  "value10" => null
  "value11" => null
   "value12" => 201912
]

Child 1 Normalized Format:

[▼
  "fromYear" => "2020"
  "value01" => 202001
  "value02" => 202002
  "value03" => 202003
  "value04" => 202004
  "value05" => 202005
  "value06" => 202006
  "value07" => 202007
  "value08" => 202008
  "value09" => 202009
  "value10" => 202010
  "value11" => 202011
  "value12" => null
]

Child 2-11 Normalized Format:

null

[xabbuh]

I think having an example application that shows the problem would make it easier to judge whether or not this is a bug.

[spackmat]

@xabbuh and here it is: https://github.com/spackmat/symfony_issue_36469

It is really quite a simple one. You can even replace my transformer code with return [] and still get six collection items inside the formView.

[xabbuh]

The reason you see this behaviour is the ResizeFormListener that is used inside the CollectionType. It listens on the PRE_SET_DATA event and adds child forms according to the number of items in your data structure. Your data transformer is only run after the event listener which explains why the norm data of the "additional" child forms is null.

Could you explain a bit what problem you are trying to solve so we can think if there is a better way to achieve that?

[spackmat]

While writing this, your answer came. So first my other observations (that are consistent with your answer):

I changed the model format to represent the correct count, but then I see that the "Normalized Format" is still ignored. Seems as the children of the collection are created before the data transformer is executed (or basing on the "Model Format").

Model Format:

[
  "year2019" => [
    "value01" => 201901,
    "value02" => 201902,
    "value03" => 201903,
  ],
  "year2020" => [
    "value01" => 202001,
    "value02" => 202002,
    "value03" => 202003,
  ],
]

Normalized Format:

[
  [
    "year" => "2019",
    "value01" => 201901,
    "value02" => 201902,
    "value03" => 201903,
  ],
  [
    "year" => "2020",
    "value01" => 202001,
    "value02" => 202002,
    "value03" => 202003,
  ],
]

Now the collection has two items, but no values are loaded. Looking deeper into the children, I see that the following data structure ist actually used:

[
  "year2020" => [
    "year" => "2019",
    "value01" => 201901,
    "value02" => 201902,
    "value03" => 201903,
  ],
  "year2020" => [
    "year" => "2020",
    "value01" => 202001,
    "value02" => 202002,
    "value03" => 202003,
  ],
]

So yes, the children are actually created on PRE_SET_DATA and the data is later loaded from the "Normalized Format", hoping best that no Transformer changed its structure meanwhile.

I solved the problem now by changing the underlying data format to be directly usable by the custom Type. That was pure luck as the format was not yet finalized. If it was, I would have had a bigger problem here.

What I wanted to solve is a problem the ModelTransformer concept IMHO is made for: I have model data that is not compatible directly to the form for it. That is in the current state not compatible to the CollectionType, when it involves changing the count of Items or when the "Model Format" is a hashed array instead of a list array (which breaks a CollectionType and now cannot be repaired by the ModelTransformer).

At least that very surprising limitation of the ModelTransformer for CollectionType should be documented! It took me a whole workday now to find out WTF is going on here, writing a reproducer and changing my code to work around that limitation.

Do you have an idea how to solve that problem, if someone cannot change the "Model Format"? Maybe transform the data on PRE_SET_DATA with a higher priority than the ResizeFormListener?

[xabbuh]

Can't you solve this by adding a transformer to the type that is used inside the CollectionType (i.e. what you configure with the entry_type option)?

[spackmat]

Unfortunately not, because the outer structure breaks the CollectionType (either by manipulating the count of collection items or because it has a string key for the collection items). You can see this in my last example (the one with the keys moved to the children: Manipulating the children doesn't fix the problem as I have to change the whole data structure to not break the CollectionType (in that case removing the keys, because the CollectionType doesn't work with a hashed array: uses the keys as indices and fails somehow over that).

[carsonbot]

Hey, thanks for your report!
There has not been a lot of activity here for a while. Is this bug still relevant? Have you managed to find a workaround?

[spackmat]

No workaround was found so far, but I could solve the problem by not doing the wanted thing as described in my last post. So state is unchanged:

At least that very surprising limitation of the ModelTransformer for CollectionType should be documented!