"Bad" SMC example doesn't actually "fail spectacularly"

[JohnGoertz]

Sequential Monte Carlo:
https://docs.pymc.io/en/stable/pymc-examples/examples/samplers/SMC2_gaussians.html:

Issue description

Ironically, this issue is about something working better than expected. The SMC example notebook has a Kill your darlings section intended to demonstrate how SMC can perform poorly with high dimensionality. However, for me, it performs basically just fine.

Expected output

4-D example online:

40-D example online:

Observed output

Note that, for better or for worse, I observed the output of the 4-D to be worse than what's online (or, at least, less smooth), while the 40-D is better than what's online.

4-D example for me:

40-D example for me:

40-D ESS summary for me:

Proposed solution

The notebook was last run with PyMC v3.9.1. My environment has:
PyMC v3.11.4
Arviz v0.11.2
Theano v1.1.2

Maybe SMC has been improved in the past few versions, and this problem needs to be made harder?

[ricardoV94]

Good catch.

This is because in v3.11.x SMC no longer defaults to a Metropolis kernel, but to an Independent-Metropolis Kernel. The metropolis kernel will be reintroduced in the next major release of PyMC (version 4), so this notebook can probably be updated to "re-fail spectacularly" then.

I suspect the transition from Metropolis to Independent Metropolis kernel happened between the notebook version and v.3.11.x

CC @aloctavodia

[JohnGoertz]

That's really interesting that there's such a big difference between the two. Perhaps the notebook should be updated at that point to indicate that switching from the (then-default) Metropolis to Independent-Metropolis can recover performance, and maybe a brief discussion as to when one would be preferable over the other?

[OriolAbril]

I would also recommend explicitly setting the kernel in addition to the note. AFAIK, pymc defaults should not be expected to be stable, as they will change and adapt in order to always reflect best practices, so things that depend on a specific initiallization, sampler or kernel should use that explicitly to avoid being version depending.

[ricardoV94]

The thing is that the old method was completely removed in V3

[aloctavodia]

Hi @JohnGoertz thanks for reporting this issue. Maybe the PyMC motto should be "better than expected", haha. Joking aside, I have updated the notebook example.

So far in our experiments we have not found an example where the Metropolis kernel performs better than the Independent-Metropolis Kernel. The main reason to re-introduce the Metropolis kernel in PyMC (version 4) was to make it easier to run further experiments comparing these two kernels.