read_namespaced_secret treat empty string as null or None

[AndyEWang]

Following is my secret.

cat test.yaml

apiVersion: v1
kind: Secret
metadata:
name: mysecret
type: Opaque
data:
username: YWRtaW4=
password: ""

kubectl create -f test.yaml

secret/mysecret created

The yaml output is:

kubectl get secret mysecret -o yaml

apiVersion: v1
data:
password: ""
username: YWRtaW4=
kind: Secret
metadata:
creationTimestamp: "2019-07-30T02:30:27Z"
name: mysecret
namespace: default
resourceVersion: "5797380"
selfLink: /api/v1/namespaces/default/secrets/mysecret
uid: fdeb42a3-b271-11e9-adf5-fa163e7bbbb1
type: Opaque

The json output is:

kubectl get secret mysecret -o json

{
"apiVersion": "v1",
"data": {
"password": "",
"username": "YWRtaW4="
},
"kind": "Secret",
"metadata": {
"creationTimestamp": "2019-07-30T02:30:27Z",
"name": "mysecret",
"namespace": "default",
"resourceVersion": "5797380",
"selfLink": "/api/v1/namespaces/default/secrets/mysecret",
"uid": "fdeb42a3-b271-11e9-adf5-fa163e7bbbb1"
},
"type": "Opaque"
}

You can see that password is empty string("") and everything looks normal by "kubectl get"
But when using read_namespaced_secret
try:
api_response = api_instance.read_namespaced_secret(name, namespace, pretty=pretty, exact=exact, export=export)
print(api_response)
except ApiException as e:
print("Exception when calling CoreV1Api->read_namespaced_secret: %s\n" % e)

The output is:
{'api_version': 'v1',
'data': {u'password': None, u'username': 'YWRtaW4='},
'kind': 'Secret',
'metadata': {'annotations': None,
'cluster_name': None,
'creation_timestamp': None,
'deletion_grace_period_seconds': None,
'deletion_timestamp': None,
'finalizers': None,
'generate_name': None,
'generation': None,
'initializers': None,
'labels': None,
'name': 'mysecret',
'namespace': 'default',
'owner_references': None,
'resource_version': None,
'self_link': '/api/v1/namespaces/default/secrets/mysecret',
'uid': None},
'string_data': None,
'type': 'Opaque'}

It's None in the output which should be empty string("")

[roycaihw]

/assign @yliaog

[yliaog]

i think the issue is due to "print(api_response)", i.e., the api_response is correct, you just need to have a smarter printer than can print out the api_response in the correct way.

[AndyEWang]

It's not related with print.
To dump the api_response to a file.
json.dump(api_response.to_dict(), f)

You can see the output that the empty string becomes null
This isn't expected.

cat /tmp/test |python -m json.tool

{
"api_version": "v1",
"data": {
"password": null,
"username": "YWRtaW4="
},
"kind": "Secret",
"metadata": {
"annotations": null,
"cluster_name": null,
"creation_timestamp": null,
"deletion_grace_period_seconds": null,
"deletion_timestamp": null,
"finalizers": null,
"generate_name": null,
"generation": null,
"initializers": null,
"labels": null,
"name": "mysecret",
"namespace": "default",
"owner_references": null,
"resource_version": null,
"self_link": "/api/v1/namespaces/default/secrets/mysecret",
"uid": null
},
"string_data": null,
"type": "Opaque"
}

[yliaog]

json.dump output null, the following snippet would print out empty string. so the problem is not with the api_response, but with the printer.

==========================================================
d=api_response.to_dict()
d.update((k,'') for k,v in d.items() if v is None)
print d

[melwynjensen]

@yliaog Understood that None is changed to Null hence update would help to come out of this issue, but then why api_response was converted from "" -> None in first place? it should have retained "" by default right? cause d.update is just an unnecessary workaround which should be done on top now.

[roycaihw]

I couldn't reproduce the issue with a 10.0.0 client against Kubernetes HEAD. I tend to agree with @yliaog that it's something with the printer. My observations:

Create the secret and read it using the python cilent:

$ kubectl version
Client Version: version.Info{Major:"1", Minor:"16+", GitVersion:"v1.16.0-alpha.2.303+7c4c029e5bbf4f-dirty", GitCommit:"7c4c029e5bbf4fcb9685b739c7ff52b823319a92", GitTreeState:"dirty", BuildDate:"2019-08-07T00:14:20Z", GoVersion:"go1.12.1", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"16+", GitVersion:"v1.16.0-alpha.2.303+7c4c029e5bbf4f-dirty", GitCommit:"7c4c029e5bbf4fcb9685b739c7ff52b823319a92", GitTreeState:"dirty", BuildDate:"2019-08-07T00:14:20Z", GoVersion:"go1.12.1", Compiler:"gc", Platform:"linux/amd64"}
$ cat secret.yaml 
apiVersion: v1
kind: Secret
metadata:
  name: mysecret
type: Opaque
data:
  username: YWRtaW4=
  password: ""
$ cat example_secret.py 
from kubernetes import client, config

def main():
    config.load_kube_config()
    configuration=client.Configuration()
    configuration.debug=True
    v1_api = client.CoreV1Api(api_client=client.ApiClient(configuration=configuration))
    api_response = v1_api.read_namespaced_secret(name='mysecret', namespace='default')
    print(api_response)

if __name__ == '__main__':
    main()
$ python example_secret.py 
2019-08-06 16:50:34,621 DEBUG Starting new HTTPS connection (1): localhost:6443
send: 'GET /api/v1/namespaces/default/secrets/mysecret HTTP/1.1\r\nHost: localhost:6443\r\nAccept-Encoding: identity\r\nContent-Type: application/json\r\nAccept: application/json\r\nUser-Agent: Swagger-Codegen/10.0.0/python\r\n\r\n'
reply: 'HTTP/1.1 200 OK\r\n'
header: Audit-Id: c3f73aef-9f70-4473-9df2-1a47177aa233
header: Content-Type: application/json
header: Date: Tue, 06 Aug 2019 23:50:34 GMT
header: Content-Length: 591
2019-08-06 16:50:34,633 DEBUG https://localhost:6443 "GET /api/v1/namespaces/default/secrets/mysecret HTTP/1.1" 200 591
{'api_version': 'v1',
 'data': {u'password': '', u'username': 'YWRtaW4='},
 'kind': 'Secret',
 'metadata': {'annotations': {u'kubectl.kubernetes.io/last-applied-configuration': '{"apiVersion":"v1","data":{"password":"","username":"YWRtaW4="},"kind":"Secret","metadata":{"annotations":{},"name":"mysecret","namespace":"default"},"type":"Opaque"}\n'},
              'cluster_name': None,
              'creation_timestamp': datetime.datetime(2019, 8, 6, 22, 53, 1, tzinfo=tzutc()),
              'deletion_grace_period_seconds': None,
              'deletion_timestamp': None,
              'finalizers': None,
              'generate_name': None,
              'generation': None,
              'initializers': None,
              'labels': None,
              'managed_fields': None,
              'name': 'mysecret',
              'namespace': 'default',
              'owner_references': None,
              'resource_version': '279',
              'self_link': '/api/v1/namespaces/default/secrets/mysecret',
              'uid': '6d2ac1f3-697c-4bb3-967b-0c44ca64cfe9'},
 'string_data': None,
 'type': 'Opaque'}

Interestingly, if I curl the apiserver directly, I see the password being null:

$ curl -v -H "Accept: application/json" localhost:8001/api/v1/namespaces/default/secrets/mysecret
> GET /api/v1/namespaces/default/secrets/mysecret HTTP/1.1
> Host: localhost:8001
> User-Agent: curl/7.64.0
> Accept: application/json
> 
< HTTP/1.1 200 OK
< Audit-Id: b965887f-cee4-4925-bf65-31337ceff52e
< Content-Length: 686
< Content-Type: application/json
< Date: Wed, 07 Aug 2019 00:26:35 GMT
< 
{
  "kind": "Secret",
  "apiVersion": "v1",
  "metadata": {
    "name": "mysecret",
    "namespace": "default",
    "selfLink": "/api/v1/namespaces/default/secrets/mysecret",
    "uid": "2e4390d0-3684-4d4d-be7f-7c5dc13e68ab",
    "resourceVersion": "276",
    "creationTimestamp": "2019-08-07T00:15:07Z",
    "annotations": {
      "kubectl.kubernetes.io/last-applied-configuration": "{\"apiVersion\":\"v1\",\"data\":{\"password\":\"\",\"username\":\"YWRtaW4=\"},\"kind\":\"Secret\",\"metadata\":{\"annotations\":{},\"name\":\"mysecret\",\"namespace\":\"default\"},\"type\":\"Opaque\"}\n"
    }
  },
  "data": {
    "password": null,
    "username": "YWRtaW4="
  },
  "type": "Opaque"
* Connection #0 to host localhost left intact

However from apiserver audit log, the json payload is an empty string:

$ cat /tmp/kube-apiserver-audit.log | grep mysecret | tail -n 1
{"kind":"Event","apiVersion":"audit.k8s.io/v1","level":"RequestResponse","auditID":"b965887f-cee4-4925-bf65-31337ceff52e","stage":"ResponseComplete","requestURI":"/api/v1/namespaces/default/secrets/mysecret","verb":"get","user":{"username":"system:admin","groups":["system:masters","system:authenticated"]},"sourceIPs":["127.0.0.1"],"userAgent":"curl/7.64.0","objectRef":{"resource":"secrets","namespace":"default","name":"mysecret","apiVersion":"v1"},"responseStatus":{"metadata":{},"code":200},"responseObject":{"kind":"Secret","apiVersion":"v1","metadata":{"name":"mysecret","namespace":"default","selfLink":"/api/v1/namespaces/default/secrets/mysecret","uid":"2e4390d0-3684-4d4d-be7f-7c5dc13e68ab","resourceVersion":"276","creationTimestamp":"2019-08-07T00:15:07Z","annotations":{"kubectl.kubernetes.io/last-applied-configuration":"{\"apiVersion\":\"v1\",\"data\":{\"password\":\"\",\"username\":\"YWRtaW4=\"},\"kind\":\"Secret\",\"metadata\":{\"annotations\":{},\"name\":\"mysecret\",\"namespace\":\"default\"},\"type\":\"Opaque\"}\n"}},"data":{"password":"","username":"YWRtaW4="},"type":"Opaque"},"requestReceivedTimestamp":"2019-08-07T00:26:35.137570Z","stageTimestamp":"2019-08-07T00:26:35.138995Z","annotations":{"authorization.k8s.io/decision":"allow","authorization.k8s.io/reason":""}}

@AndyEWang Could you try the up-to-date versions for the server and the client with DEBUG log enabled, and see if the issue is reproducible?

[AndyEWang]

@roycaihw
It's caused by parameter "pretty".
When pretty is "true", the output is None
When pretty is "false", the output is empty string("")
Do you know why pretty leads to different output?

[roycaihw]

This is in the json-iterator that apiserver uses and it's fixed in json-iterator/go#371

I sent kubernetes/kubernetes#81096 to pick up the fix in apiserver

/close
as it's not a client issue

[k8s-ci-robot]

@roycaihw: Closing this issue.

In response to this:

This is in the json-iterator that apiserver uses and it's fixed in json-iterator/go#371

I sent kubernetes/kubernetes#81096 to pick up the fix in apiserver

/close
as it's not a client issue

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.