Missing AdaDeltaOptimizer

[tcwicks]

Is it anyway possible to please get AdaDeltaOptimizer implemented.

The reason I ask is because for Variational Auto Encoders (Essential for dimensionality reduction) Adam does not do a very good job and mostly fails. Reducing Beta1 to 0.1 and Beta2 to 0.199 kind of half works with Adam but still not that well. However AdaDelta just simply works.

I would port this but I have no idea how to write the Gen_ops part.
Please and Thanks

[Oceania2018]

We would love to add more Optimizers.
Can you provide the AdaDeltaOptimizer UnitTest both in C# and Python ?
Please show us which specific part you can't get it done.

[tcwicks]

This is the Tensorflow python implementation:

https://github.com/tensorflow/tensorflow/blob/ba63d9082a2265da91ec4daefecfa4cd47fcf07f/tensorflow/python/training/adadelta.py

I was looking at how Adam was implemented in SciSharp Tensorflow and it is pretty straight forward to port it. Except for one small bit:

Which is this piece: In the python code there are two calls to training_ops

return training_ops.apply_adadelta(........
and
training_ops.sparse_apply_adadelta(.........

The equivalent calls for apply sparse and dense would need to be implemented (looking at Adam in Sci Sharp) in:
Tensorflow.gen_training_ops

Every time I've tried to port something over I get stuck at this last piece in gen_training_ops
I just don't quite get how to implement the gen_training_ops piece. Happy to contribute if I just know howto implement this last bit.

[Oceania2018]

You can find the file from

It would be helpful if you could provide UnitTest in C# and Python.