sampler stats

[mjhajharia]

This notebook uses InferenceData and ArviZ for plotting. Explains multi-step sampler_stats as well.
Changed sampler_stats description according to - https://arviz-devs.github.io/arviz/schema/schema.html#sample-stats

Addresses issue #46

[review-notebook-app]

Check out this pull request on 

See visual diffs & provide feedback on Jupyter Notebooks.

---

Powered by ReviewNB

[mjhajharia]

deleted PR Samplerstats(WIP) #94 and created this, will take care of comments made there and make a commit, sorry about the inconvenience, i tried resetting to an initial commit and that messed up the previous PR so started it again

[mjhajharia]

I tried trace.sample_stats["tree_depth"].plot(row="chain", ls="none", marker=".", alpha=.3)
but I think because of the number of points or something, it still resembles a line plot, visually looks way better though.

@OriolAbril also can you mark the useless plot again, since that pr has 0 commits i cannot see the location of those comments

I'm not very sure if this plot is better than the previous one:

previous being:

finally im so sorry i made these comments on the issue you referenced the PR in, instead of here, deleted those and put them here

[OriolAbril]

I tried trace.sample_stats["tree_depth"].plot(row="chain", ls="none", marker=".", alpha=.3)
but I think because of the number of points or something, it still resembles a line plot, visually looks way better though.

Let's go with col instead of hue then, but keep the alpha. tree_depth is an integer, and it has one value per draw, as there isn't much variation, it generates these visually looking flat lines, but you can see that on 1 there are some stray points. I think the scatterplot with col will look good enough.

The plot step_size bar is the one I find useless, we should just remove it and be done with it.

The comment about using plot_posterior was for the histograms for the acceptance rates, that you have already updated, not for this step size plot.

[OriolAbril]

dims and coords seem to have been ignored, could be a bug in ArviZ. I'll look into that. Do change this for the plot anyway though, even if at least for now the labels are wrong. Using plot_posterior like we did above should work here too.

Reply via ReviewNB

[mjhajharia]

okayy, could it be because we didn't refer to dims in step1 and step2

[OriolAbril]

ArviZ ignores the dimensions for sample stats: https://github.com/arviz-devs/arviz/blob/main/arviz/data/io_pymc3.py#L323 it may be to avoid name collisions though.

[mjhajharia]

do you think that could use some fixing, maybe an informative warning or error could be given during a name collision. Because in real time models and stuff diagnostics running well, following general practices of arviz/pymc3 seems to be actually very important

[OriolAbril]

Yes, we could fix that in ArviZ and have it use coords and dims also in the sample stats, either with a try except or checking things more carefully. But this is very low priority, compound steps are not that common and we have to make sure to implement that in a way that it never backfires.

[mjhajharia]

made some changes(experimental)

[OriolAbril]

@michaelosthege can you explain what are perf_counter_diff , process_time_diff and perf_counter_start? Or point us to an issue or PR about them?

[mjhajharia]

@michaelosthege can you explain what are perf_counter_diff , process_time_diff and perf_counter_start? Or point us to an issue or PR about them?

hi, maybe this comment wasn't notified or something because it was in the reviewnb thing,
#95 (comment) , but I found these descriptions in the NUTS sampler documentation

[mjhajharia]

i think it still doesn't open in the mobile app, so im copying the comment here:

@OriolAbril no problem at all! also I just checked and the items ive added are the exact same, I went through the code for sample_stats_to_array here , that clarified it for me. thanks anyway this looks like a helpful comment.

this is from the current commit on the notebook, in case you wanted to see(only tune needs to go i think)

[mjhajharia]

here is the relevant documentation

[OriolAbril]

Yeah, I am seeing it now, sorry about that. Thanks for looking into this.

[OriolAbril]

You are right that now all the relevant variables are present, only tune has to be removed.

Regaring the accept variable from metropolis, I am not sure it "deserves" so much analysis, but I still don't understand what it is. The analysis is great though

[mjhajharia]

You are right that now all the relevant variables are present, only tune has to be removed.

Regaring the accept variable from metropolis, I am not sure it "deserves" so much analysis, but I still don't understand what it is. The analysis is great though

thanks!! and yeah that is somewhat weird, all i could find in metropolis.py was:
accept = self.delta_logp(q, q0)

Also tune is a stat in metropolis.py but I don't see it in sample_stats, so I'm a little confused about whether it should be needed or not. I think it got removed because of the compound step thing, if it's there in a single model sampling for metropolis dist. then we might need it?

[mjhajharia]

I looked at the original notebook once again, and possibly the reason why they chose to show accept, was just that it was common in both the samplers, which might not be a good enough reason for an elaborate plot which also has outliers. I might be entirely wrong here, I'm just speculating.

[OriolAbril]

and yeah that is somewhat weird, all i could find in metropolis.py was:
accept = self.delta_logp(q, q0)

I took a look at that, now it make sense. In metropolis one generates a proposal q and then compares it's probability to the current sample q0 with alpha = P(q) / P(q0). If alpha > 1, that is, q has higher probability than q0 the proposal is always accepted, otherwise alpha is in the range 0, 1 so sample from the uniform distribution u is generated, if u>alpha the proposal is rejected, accepted otherwise. accept is alpha which is also sometimes called acceptance ratio

I looked at the original notebook once again, and possibly the reason why they chose to show accept, was just that it was common in both the samplers, which might not be a good enough reason for an elaborate plot which also has outliers. I might be entirely wrong here, I'm just speculating.

yes, this sounds right, our goal here should probably be to show that when using a compound step, the common variables in sample_stats will get an extra dimension, one per sampler.

[mjhajharia]

I took a look at that, now it make sense. In metropolis one generates a proposal q and then compares it's probability to the current sample q0 with alpha = P(q) / P(q0). If alpha > 1, that is, q has higher probability than q0 the proposal is always accepted, otherwise alpha is in the range 0, 1 so sample from the uniform distribution u is generated, if u>alpha the proposal is rejected, accepted otherwise. accept is alpha which is also sometimes called acceptance ratio

yeah, that makes sense, thanks for explaining, i just started reading up about it and noticed acceptance rate was important for metropolis hastings.

yes, this sounds right, our goal here should probably be to show that when using a compound step, the common variables in sample_stats will get an extra dimension, one per sampler.

In that case, the last plot is unnecessary, but possibly showing the range of accept on both the samplers, would make some sense?

[OriolAbril]

acceptance rate was important for metropolis hastings.

In metropolis it's acceptance ratio which is very different from the acceptance rate in hmc/nuts

In that case, the last plot is unnecessary, but possibly showing the range of accept on both the samplers, would make some sense?

Yes, I think that showing how to work with sample stats that have multiple steps is useful and won't be explained anywhere else probably

[mjhajharia]

In metropolis it's acceptance ratio which is very different from the acceptance rate in hmc/nuts

yes, my bad, ive been looking at nuts doc. over and over for the stats thing sort of fixated in my head.

Yes, I think that showing how to work with sample stats that have multiple steps is useful and won't be explained anywhere else probably

yes we can do that, also I think a detailed explanation could be added here as well - compound step notebook

[OriolAbril]

yes we can do that, also I think a detailed explanation could be added here as well - compound step notebook

good catch, I think I haven't gotten to create the issue for that notebook, I don't remember ever reading it until now.

[mjhajharia]

yes we can do that, also I think a detailed explanation could be added here as well - compound step notebook

good catch, I think I haven't gotten to create the issue for that notebook, I don't remember ever reading it until now.

thanks! I haven't come across an issue on that, if you want I can make one, or I can directly make a draft PR too, whichever seems better?

[OriolAbril]

I will create an issue now

[mjhajharia]

I will create an issue now

sure!

[OriolAbril]

can you restart and run all on the notebook? This should get rid of this warning in the last cell due to having executed it twice:

The watermark extension is already loaded. To reload it, use:
%reload_ext watermark

[mjhajharia]

@OriolAbril done, it can be merged now