Text and SQLite backends for PyMC3 (update) #500

kyleam · 2014-03-01T00:04:57Z

These changes are based on the discussion in #449. Please let me know your thoughts.

twiecki · 2014-03-01T15:40:18Z

======================================================================

ERROR: Failure: ImportError (No module named mock)

----------------------------------------------------------------------

Traceback (most recent call last):

File "/home/travis/miniconda/envs/testenv/lib/python2.7/site-packages/nose/loader.py", line 413, in loadTestsFromName

addr.filename, addr.module)

File "/home/travis/miniconda/envs/testenv/lib/python2.7/site-packages/nose/importer.py", line 47, in importFromPath

return self.importFromDir(dir_path, fqname)

File "/home/travis/miniconda/envs/testenv/lib/python2.7/site-packages/nose/importer.py", line 94, in importFromDir

mod = load_module(part_fqname, fh, filename, desc)

File "/home/travis/build/pymc-devs/pymc/pymc/tests/test_ndarray_backend.py", line 6, in <module>

import mock

ImportError: No module named mock

Probably need to add mock to the travis build.

kyleam · 2014-03-01T16:32:10Z

Thanks. It's pre-installed [1] on travis, but I didn't consider that
we're using conda now.

[1] http://docs.travis-ci.com/user/languages/python/#Pre-installed-packages

This is only needed for python 2 because mock is in stdlib for python 3 (unittest.mock).

This commit contains a new backend for sampling and selecting values. Non-backend files have been changed to work with the new backend. This commit also merges the `sample` and `psample` functions. `sample` now takes a keyword argument `njobs`, and if this is over one, the multiprocessing version is used.

twiecki · 2014-03-01T20:05:05Z

Is this good to merge then? I know @jsalvatier reviewed this and it certainly looks high quality.

kyleam · 2014-03-01T20:23:42Z

Is this good to merge then? I know @jsalvatier reviewed this and it
certainly looks high quality.

Sorry, I should have indicated that when I opened up the new PR. I think
it'd be nice to give the new version some time for review and
discussion.

fonnesbeck · 2014-03-05T15:57:15Z

I notice that if you try running another chain using an existing backend, the chain variable does not get incremented in the backend (nor does the draw get reset to zero). It would be nice to have the backend check for the highest chain number before sampling, so that we can arbitrarily go back and add chains to a database without confusing which chain the samples came from.

Great work, BTW!

fonnesbeck · 2014-03-05T16:14:28Z

Never mind. Now I see that the chain variable is set via sample. That's probably the best approach.

fonnesbeck · 2014-03-19T16:56:24Z

After running a few tests, I think this looks really good. A couple of things:

we should add an example (or modify an existing one) that uses the SQLite backend
it might be nice to have a shortcut for using the backend with a default name. At present we have to pass trace = pymc.backends.SQLite('my_backend') to the sample call. Might be nice to be able to simply have trace="sqlite" do that for you, with a default name like "MCMC" or "model" for the database name.

kyleam · 2014-03-19T17:38:28Z

Thanks for the suggestions. I'll push an update soon.

kyleam · 2014-03-20T16:51:16Z

OK, I've pushed updates that incorporate your suggestions.

kyleam · 2014-03-20T18:50:45Z

As the Travis failures show, I only ran a subset of the tests locally.

kyleam · 2014-03-20T20:20:00Z

The Travis issues should be fixed now.

I'm moving this under main so that it doesn't run with "test_examples". This could be set up like the other examples, with a run definition that allows for a short version, but I'd prefer not to for a couple of reasons. 1. Everything aside from the SQLite backend is the same as "hierarchical.py", so it isn't testing much more for the time added to the run. 2. This results in an SQLite file, so a cleanup should be added somewhere if it is run with "test_examples".

jsalvatier · 2014-04-06T23:52:58Z

pymc/tests/test_backend_dump_load.py

+            njobs = 1
+
+        data = np.random.normal(size=(2, 20))
+        model = pm.Model()


I would just get a stock model from tests/models.py or at least move this model into there.

Good point.

jsalvatier · 2014-04-07T00:04:31Z

pymc/tests/test_base_backend.py

+
+    def test_multitrace_init_unique_chains(self):
+        trace0 = mock.Mock()
+        trace0.chain = 0


is it necessary to set trace0.chain? That seems undesirable. In general it would be nice to mostly test external behavior and internal behavior less.

jsalvatier · 2014-04-07T00:59:36Z

Kyle, I want to thank you for your hard work and patience. You've put a lot of effort into this and it shows. I really like the external interface to the trace objects. Its simple and functional.

I think I might have opinions about how the backends are implemented, but its clear to me that I'm not going to put in the effort to help make those changes right now. So I'm in favor of merging this soon.

kyleam · 2014-04-07T03:18:10Z

(Since several of your comments address the same thing, I'll just respond in a general comment.)

I think there's a trade-off between having isolated/fast tests and relying too much on internal details and mocking. Based on your comments (and because my tests are the first place mocking occurs in the test suite), I'd guess we have different preferences on this. I've tried to also add higher level tests (like those in the dump/load and selection tests). For now, I'd prefer to keep these unit tests and just extend the higher level tests as desired. The unit tests add a very small amount of time to the total test suite, and they can be deleted later if they become problematic.

kyleam · 2014-04-07T03:18:54Z

I'll incorporate some of your suggestions, and then rebase this (since it will won't apply cleanly to recent changes in master) and open a new PR.

jsalvatier · 2014-04-08T04:55:32Z

That's true, about fast tests. Our tests are generally pretty slow.

I look forward to seeing your changes.

Include tests_require argument in setup.py

9266900

kyleam added 7 commits March 1, 2014 11:49

Include mock as test dependency

2bd8e5b

This is only needed for python 2 because mock is in stdlib for python 3 (unittest.mock).

Add Text backend

3a83062

Add SQLite backend

d5295a4

Add backend documentation

3c01fba

Test equality of NDArray and SQLite selections

640be84

Dump and load tests for text and SQLite

6890fe9

kyleam added 4 commits March 20, 2014 00:32

Remove unused multiprocessing from tests

764cdd1

Add missing shutil import to test

c3533e0

Clean up obsolete information in docstring

0d5c5f9

Fix hardcoded njobs argument in dump test

92d813f

kyleam added 2 commits March 20, 2014 15:59

Add shortcuts for Text and SQLite backends

5978859

Add SQLite backend example for hierarchical.py

ec92cf2

jsalvatier reviewed Apr 6, 2014
View reviewed changes

jsalvatier reviewed Apr 7, 2014
View reviewed changes

kyleam mentioned this pull request Apr 9, 2014

Text and SQLite backends (update) #527

Merged

kyleam closed this Apr 9, 2014

kyleam deleted the pymc3-backends-2 branch May 25, 2014 16:57

Uh oh!

Text and SQLite backends for PyMC3 (update) #500

Text and SQLite backends for PyMC3 (update) #500

Uh oh!

Conversation

kyleam commented Mar 1, 2014

Uh oh!

twiecki commented Mar 1, 2014

Uh oh!

kyleam commented Mar 1, 2014

Uh oh!

twiecki commented Mar 1, 2014

Uh oh!

kyleam commented Mar 1, 2014

Uh oh!

fonnesbeck commented Mar 5, 2014

Uh oh!

fonnesbeck commented Mar 5, 2014

Uh oh!

fonnesbeck commented Mar 19, 2014

Uh oh!

kyleam commented Mar 19, 2014

Uh oh!

kyleam commented Mar 20, 2014

Uh oh!

kyleam commented Mar 20, 2014

Uh oh!

kyleam commented Mar 20, 2014

Uh oh!

jsalvatier Apr 6, 2014

Choose a reason for hiding this comment

Uh oh!

kyleam Apr 7, 2014

Choose a reason for hiding this comment

Uh oh!

jsalvatier Apr 7, 2014

Choose a reason for hiding this comment

Uh oh!

jsalvatier commented Apr 7, 2014

Uh oh!

kyleam commented Apr 7, 2014

Uh oh!

kyleam commented Apr 7, 2014

Uh oh!

jsalvatier commented Apr 8, 2014

Uh oh!

Uh oh!