Update client to support a different project_id to run jobs #144
Conversation
This supports authenticate to 1 project_id but run jobs in a different project_id.
| job_reference: json of job_reference | ||
| """ | ||
| job_reference = { | ||
| "projectId": self.project_id, |
There was a problem hiding this comment.
Shouldn't this use the overridden project_id if it's passed into the function?
There was a problem hiding this comment.
Hi, @tylertreat,
Correct, it just override a different project_id if it's passed other than the self.project_id. This is to resolve the problem when the service account does not have project permission, which means it cannot authenticate to the project. There is a use case at my company like this:
- a service account is created, but all jobs are submitted to a default billing
project_id. - This service account does not have project permission for other projects due to many sensitive datasets. In this case, the service account cannot authenticate to those projects. So the service account only has permission to access only a few datasets in that project if the service account is added to those datasets with view or edit permisson.
- However, Google supports this use case which is the job can be submitted to a billing project for billing purposes and tracking jobs, but it can process data in another dataset in another project which it is shared.
In short, this resolves the use case where service account can submit and run jobs even in another project that it does not have project permission but only dataset permission.
Hope it makes sense. I know it is a bit complicated, but I know there are a few companies have these restrictions to protect their data. Especially if you work with third parties data where they only share the dataset with you. But in other to run the jobs, you have to submit jobs to a different billing project_id.
Best,
Tuan
There was a problem hiding this comment.
Yes, sorry, I understand the overall changes and it makes sense to me. I was specially referring to _get_job_reference and why it's hardcoded to use self.project_id instead of passing in a project_id and calling _get_project_id(project_id) like the rest of your changes?
There was a problem hiding this comment.
Oh, because the _insert_job always submit jobs to self.project_id, which is the billing project. Then jobReference.projectId will always be self.project_id. That is why _get_job_reference is hardcoded using the self.project_id. Hope it makes sense!
There is a use case where clients want to authenticate and enable billing to 1 project but run all the async query, load, export, delete jobs in a different project if specified. The default is self.project_id.