[MRG] FIX: fix lsml inversion

[wdevazelhes]

Fixes #202

[perimosocordiae]

Looks good. It's possible that this will mask some issues from users that provide a garbage matrix as the prior, but that's probably okay.

[wdevazelhes]

somehow the covariance matrix in the previous example was not exactly singular so I built the example by hand, and it is indeed truly singular and exhibited another place where we should use the pseudo inverse

[bellet]

I am not sure that this is a good fix, because logdet divergence is finite only if the matrix is (strictly) positive definite. Hence working the whole time with singular matrices and pseudo-inverses looks like deviating quite a bit from the algorithm.

I think a better fix may be simply to return an error when the prior is not positive definite. When this happens with the covariance, the identity can be used (this could be done in #180)

[wdevazelhes]

I am not sure that this is a good fix, because logdet divergence is finite only if the matrix is (strictly) positive definite. Hence working the whole time with singular matrices and pseudo-inverses looks like deviating quite a bit from the algorithm.

I think a better fix may be simply to return an error when the prior is not positive definite. When this happens with the covariance, the identity can be used (this could be done in #180)

That's right, I didn't think of that...
I noticed that there is the same problem for SDML, and I'm thinking about this case: say the user provides a prior that is not PSD. But the emp_cov matrix could be PSD no? In this case should we still forbid an indefinite prior ? I guess so because it does not makes sense given what is a prior ? (it should be a mahalanobis metric)

EDIT: I got confused with PD and PSD: I guess this was the question
I noticed that there is the same problem for SDML, and I'm thinking about this case: say the user provides a prior that is not PD. But the emp_cov matrix could be PD no? In this case should we still forbid an indefinite prior ?
And after looking at SDML, I just realized that we need the inverse of the prior anyway which can be done only if it is definite, so the answer is we should also forbid the prior to be indefinite there

[wdevazelhes]

Since in the end we will use init='identity' for the examples, I guess we don't need to merge this PR quicker, and the work will be done in PR #195 , so I'm closing the PR

[bellet]

I am not sure that this is a good fix, because logdet divergence is finite only if the matrix is (strictly) positive definite. Hence working the whole time with singular matrices and pseudo-inverses looks like deviating quite a bit from the algorithm.

I think a better fix may be simply to return an error when the prior is not positive definite. When this happens with the covariance, the identity can be used (this could be done in #180)

Let me correct my own claim. While the LogDet divergence is indeed finite only for pairs of PD matrices, in LSML and other algorithms using LogDet (ITML, SDML) having a PSD (not strictly PD) prior is probably fine in most cases because the logdet term on the prior is ignored since it is constant. So this PR could be merged if you use pseudo-inverse only for the prior (but not for the gradient!).

Yet, I think it is probably safer to rule out non-PD priors for these algorithms (see my other comment in #195)