Skip to content

Reflector list should handle expired resource version from watch handler as well #1628

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 1 commit into from
Closed

Reflector list should handle expired resource version from watch handler as well #1628

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 5 additions & 1 deletion util/src/main/java/io/kubernetes/client/informer/cache/ReflectorRunnable.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -237,7 +237,11 @@ private void watchHandler(Watchable<ApiType> watch) {
}
if (eventType.get() == EventType.ERROR) {
if (item.status != null && item.status.getCode() == HttpURLConnection.HTTP_GONE) {
log.info("Watch connection expired: {}", item.status.getMessage());
log.info(
"ResourceVersion {} and Watch connection expired: {} , will retry w/o resourceVersion next time",
getRelistResourceVersion(),
item.status.getMessage());
isLastSyncResourceVersionUnavailable = true;
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@aaruna upon 410 status code, the watch handler is supposed to return and we deal w/ the RV expiration at the list call. this is working as designed. https://github.com/kubernetes/kubernetes/blob/a96000311f5beca111debc8727018e42cbb5dc79/staging/src/k8s.io/client-go/tools/cache/reflector.go#L129-L131

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Are you referring to the next list call? Because the current watch handler won't throw an ApiException and only that is handled in the catch block above. (The unit test added in this PR shows an example of this scenario)

CC @jiangzhou in case I have missed anything in the scenario above.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Are you referring to the next list call?

yeah. IIRC watch handler doesnt directly throw an ApiException but it will deal w/ error status nested in the decoded watch event.

java/util/src/main/java/io/kubernetes/client/informer/cache/ReflectorRunnable.java

Lines 227 to 237 in c580990

if (eventType.get() == EventType.ERROR) {
if (item.status != null && item.status.getCode() == HttpURLConnection.HTTP_GONE) {
log.info("Watch connection expired: {}", item.status.getMessage());
return;
} else {
String errorMessage =
String.format("got ERROR event and its status: %s", item.status.toString());
log.error(errorMessage);
throw new RuntimeException(errorMessage);
}
}

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

But that's not resetting isLastSyncResourceVersionUnavailable=true. The unit test below has an example

throw new WatchExpiredException();
} else {
String errorMessage =
Expand Down
31 changes: 31 additions & 0 deletions util/src/test/java/io/kubernetes/client/informer/cache/ReflectorRunnableTest.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,7 @@
package io.kubernetes.client.informer.cache;

import static org.junit.Assert.assertFalse;
import static org.junit.Assert.assertTrue;
import static org.mockito.Mockito.any;
import static org.mockito.Mockito.mock;
import static org.mockito.Mockito.never;
Expand Down Expand Up @@ -312,4 +313,34 @@ public void testReflectorWatchShouldRelistUponExpiredWatchItem() throws ApiExcep
.until(() -> future.isDone());
assertFalse(future.isCompletedExceptionally());
}

@Test
public void testReflectorListShouldHandleExpiredResourceVersionFromWatchHandler()
throws ApiException {
String expectedResourceVersion = "100";
when(listerWatcher.list(any()))
.thenReturn(
new V1PodList().metadata(new V1ListMeta().resourceVersion(expectedResourceVersion)));

V1Status v1Status = new V1Status();
v1Status.setMessage("dummy-error-message");
v1Status.setCode(410);
when(listerWatcher.watch(any()))
.thenReturn(new MockWatch<>(new Watch.Response("Error", v1Status)));
ReflectorRunnable<V1Pod, V1PodList> reflectorRunnable =
new ReflectorRunnable<>(V1Pod.class, listerWatcher, deltaFIFO);
try {
Thread thread = new Thread(reflectorRunnable::run);
thread.setDaemon(true);
thread.start();
Awaitility.await()
.atMost(Duration.ofSeconds(1))
.pollInterval(Duration.ofMillis(100))
.until(
() -> expectedResourceVersion.equals(reflectorRunnable.getLastSyncResourceVersion()));
assertTrue(reflectorRunnable.isLastSyncResourceVersionUnavailable());
} finally {
reflectorRunnable.stop();
}
}
}