Implement cluster model by subclassing of tensorflow.keras #16

BlackPoint-CX · 2019-09-10T18:10:56Z

Has mostly implement cluster model by subclassing of tensorflow.keras.
Still work in progress.

TODO:

Implement logic of saving and loading weights.
Implement modification of codegen in sqlflow.

tonyyang-svail

Excellent PR.

tonyyang-svail · 2019-09-10T19:04:09Z

sqlflow_models/cluster_keras.py

+        metric['ari'] = np.round(ari(y, y_pred), 5)
+        return q, metric
+
+    def cluster_train_loop(self, x, y, batch_size=256, maxiter=8000, update_interval=150, tol=0.001):


What is the usage of y? I thought the cluster model is trained on unlabeled data.

y is used for evaluating the performance of cluster model when training with labeled data.
You are right. It is really necessary to remove this or setting it as None by default.

tonyyang-svail · 2019-09-10T19:06:35Z

sqlflow_models/cluster_keras_main.py

+    print(dec.display_model_info(verbose=2))
+
+    if hasattr(dec, 'cluster_train_loop'):
+        dec.cluster_train_loop(x=x, y=y,


I am wondering if it is possible to use the model.fit API to training the model, and put the early stopping strategy as a callback function? So that the code generation template in SQLFlow would be simpler.

I am entangled with this.

In classical scenario, when subclassing the keras.Model class, user only need to define the structure of model in __init__ and the forward pass in the call method. All the other operations such as compile and fit are inherited from keras.Model.

However in cluster model, the inherited fit cannot fit with such training process.
Cluster model uses auxiliary target distribution p as latent label for training and it keeps changing during training. The inherited fit always requires static y. Based on these I used train_on_batch instead of fit.

q, metric = self.evaluate(x, y) p = self.target_distribution(q) ... loss = self.train_on_batch(x=x[idx], y=p[idx])

Yancey0623 · 2019-09-11T06:18:56Z

sqlflow_models/clsuter_keras_output.txt

@@ -0,0 +1,520 @@
+/Users/didi/Develop/VirtualEnvs/virtualenv_3.6.5/bin/python3.6 /Users/didi/Develop/PycharmProjects/temp_requirements/src/order_by_project/sqlflow/clustering_model/version_03/dec_demo_03.py


Please remove this log file.

OK, got it.

Yancey0623 · 2019-09-11T06:19:56Z

sqlflow_models/cluster_keras.py

+                 pretrain_initializer='glorot_uniform',
+                 loss=None):
+        """
+        Implement cluster model mostly based on DEC.


Add a link to DEC?

I am not sure that the cluster model is DEC but they are really close.
I am entangled with whether this link will mislead users.

Yancey0623 · 2019-09-11T06:39:13Z

sqlflow_models/cluster_keras.py

+        self.get_layer(name='clustering').set_weights([self.kmeans.cluster_centers_])
+        self.display_model_info()
+        index, loss, p = 0, 0., None
+        y_pred_last = self.kmeans.fit_predict(self.encoded_input)


This line is redundancy, L175 can return the y_pred_last.

Got it. Will remove it.

Yancey0623

As the discussion with @wenjing , the output prediction result group by group_id, so maybe we need to implement a function sqlflow_predict() function that outputs the result which group by group_id.

Yancey0623 · 2019-09-16T09:08:56Z

sqlflow_models/cluster_keras_main.py

+
+
+@timelogger
+def train_evaluate(datasource='mnist'):


Please remove train_evaluate function in the model definition file, and add a unit test in the tests folder.

Yancey0623 · 2019-09-17T07:17:19Z

tests/test_deep_embedding_cluster.py

+        self.label = [0 for _ in range(50)] + [1 for _ in range(50)]
+        feature_columns = [tf.feature_column.numeric_column(key) for key in
+                           self.features]
+        self.model = sqlflow_models.DNNClassifier(feature_columns=feature_columns)


Maybe we need to test sqlflow_models.DeepEmbeddingCluster here.

Maybe we need to test sqlflow_models.DeepEmbeddingCluster here.

Have added in new commit. Tks for remind.

…alling train_on_batch.

Yancey0623

The CI failed logs:

    import sqlflow_models
sqlflow_models/__init__.py:4: in <module>
    from .deep_embedding_cluster import DeepEmbeddingClusterModel
sqlflow_models/deep_embedding_cluster.py:19: in <module>
    from sklearn.cluster import KMeans
E   ModuleNotFoundError: No module named 'sklearn'

You can fix it by adding the dependency in https://github.com/sql-machine-learning/models/blob/develop/setup.py#L24

Yancey0623 · 2019-09-17T10:08:27Z

sqlflow_models/deep_embedding_cluster.py

+        self.y_pred_last = self.kmeans.fit_predict(self.encoded_input)
+        print('{} Done init centroids by k-means.'.format(datetime.now()))
+
+    def cluster_train_loop(self, x):


Maybe we can make cluster_train_loop to be more general:

def sqlflow_trai_loop(self, dataset, epochs, verbos): ...

Maybe we can make cluster_train_loop to be more general:

def sqlflow_trai_loop(self, dataset, epochs, verbos): ...

Actually I have thought about this.
epochs is always used to define how many times to train on the whole dataset. However in this case, the auxiliary target distribution p needs to be updated after update_interval iterations (update_interval always smaller than the number of batches in one epoch). Since the pre-train process can fetch parameter pretrain_epochs from construct function, and pre_train should be consider as sub-train-process, I did not add epochs as parameter here.
verbose can be add here for control to display more details. I can add this later.

Thanks, I see that epochs is not necessary for this cause, but we need to support the custom train loop function by editing template_tf.go in SQLFlow, and I think epochs is useful for the most of the scenes, so maybe we can to add epochs argument with a default value.

…ion cluster_train_loop to sqlflow_train_loop

…sqlflow.

…rm of custom model.

Yancey0623

LGTM, thanks for the excellent PR!

alfredchenxiang added 2 commits September 7, 2019 22:38

WIP: Implement cluster

fb10868

WIP : Implement cluster by subclassing of tensorflow.keras

772adb0

tonyyang-svail reviewed Sep 10, 2019

View reviewed changes

Yancey0623 reviewed Sep 11, 2019

View reviewed changes

Yancey0623 previously approved these changes Sep 11, 2019

View reviewed changes

Yancey0623 mentioned this pull request Sep 11, 2019

WIP: Implement cluster #15

Closed

Yancey0623 reviewed Sep 16, 2019

View reviewed changes

Yancey0623 mentioned this pull request Sep 16, 2019

[TODO List] run Cluster Model in SQLFlow sql-machine-learning/sqlflow#768

Closed

3 tasks

Implement Deep Embedding Cluster Model.

62dffbc

BlackPoint-CX dismissed Yancey0623’s stale review via 62dffbc September 17, 2019 07:09

Yancey0623 reviewed Sep 17, 2019

View reviewed changes

Update testcase of deep_embedding_cluster ; Change split logic when c…

4eda695

…alling train_on_batch.

Yancey0623 reviewed Sep 17, 2019

View reviewed changes

alfredchenxiang added 5 commits September 17, 2019 18:15

Update setup.py for requirement of package scikit-learn; Rename funct…

a4c456b

…ion cluster_train_loop to sqlflow_train_loop

Update setup.py for specified version of scikit-learn.

1111789

Update requriments of package : numpy, pandas

38c3fcc

Update setup.py : Change version of tensorflow to 2.0.0b1, same with …

66c3aa6

…sqlflow.

Add parameter epoch and verbose for sqlflow_train_loop to meet the fo…

07a0fc8

…rm of custom model.

BlackPoint-CX changed the title ~~WIP : Implement cluster model by subclassing of tensorflow.keras~~ Implement cluster model by subclassing of tensorflow.keras Sep 17, 2019

Yancey0623 approved these changes Sep 17, 2019

View reviewed changes

typhoonzero approved these changes Sep 17, 2019

View reviewed changes

weiguoz merged commit 73adf6a into sql-machine-learning:develop Sep 17, 2019

		@@ -0,0 +1,520 @@
		/Users/didi/Develop/VirtualEnvs/virtualenv_3.6.5/bin/python3.6 /Users/didi/Develop/PycharmProjects/temp_requirements/src/order_by_project/sqlflow/clustering_model/version_03/dec_demo_03.py

Implement cluster model by subclassing of tensorflow.keras #16

Implement cluster model by subclassing of tensorflow.keras #16

Uh oh!

Conversation

BlackPoint-CX commented Sep 10, 2019

Uh oh!

tonyyang-svail left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

BlackPoint-CX Sep 11, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Yancey0623 left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Yancey0623 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Yancey0623 Sep 17, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Yancey0623 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

BlackPoint-CX Sep 11, 2019 •

edited

Loading

Yancey0623 left a comment •

edited

Loading

Yancey0623 Sep 17, 2019 •

edited

Loading