Consider splitting off rules core to building and and testing scala from those that arent

[softprops]

We've been bit a few times on upgrade paths due to transient dependencies of rules within this repo I feel are not core to scala support.

Specifically protobuf dependencies are always hard for us to manage.

This could be remedied by keeping rules like scala library in a repo separate from aux rules like thrift, scalapb etc.

I understand this introduces a burden on maintainers but the reward for users feels worth it.

[johnynek]

You can feel free to make a port of this code and start separate rules, and you may be right that they can be independently run well now. Since stripe has done almost all the work on thrift and did the rewrite of the proto rules to fix the issues (recompilation, duplicate classes), I don’t think we would be that motivated to take on more work. Our scala team has 3 people and they are very taxed as is.

We need more help, and our current situation isn’t maintainable. Any contributions you can make would be welcome. In the mean time, anyone is free to use the code but I don’t see much, of any, bandwidth for external requests.

This is currently best understood as a collaboration between Wix and Stripe to help our builds, and if we can help others enough to motivate them to contribute, that would be great too.

[ittaiz]

+1 to Oscar's general statement.
@softprops Is it accurate to say that concretely your main pains are dependencies?
If so I think the main contribution that I'd appreciate is splitting off our "deps" pattern to separate toolchains. This will advance the rules and decouple you from the parts you don't care about.
How does that sound?

[ittaiz]

Btw,
I'm working (very lazily) on moving the junit/specs2 test discovery out of this repo. Both to make it more focused and to allow java users to use it.

[softprops]

Protobuf in particular yes. Thanks for the direction on the separate toolchains vs separate repo approach. I had not yet thought of that

[ittaiz]

Sure. In general Oscar and I agreed in the past about wanting to split everything to smaller toolchain. Didn’t get around to it :(

…

On Fri, 14 Jun 2019 at 15:58 Doug Tangren ***@***.***> wrote: Protobuf in particular yes. Thanks for the direction on the separate toolchains vs separate repo approach. I had not yet thought of that — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#769>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAKQQFYJ5YYN5GGHKDF32UTP2OIXHANCNFSM4HYCHNCQ> .

[beala-stripe]

Last I checked, the protobufs dep is used by more than just the proto rules: #726 (comment)

Maybe it could be factored out, but by the looks of that bazel query, it would be non-trivial.

[ittaiz]

Ok, not sure why this happened but I’m really in favor of splitting this. @softprops any chance you want to work on identifying why we have these deps?

…

On Fri, 14 Jun 2019 at 23:01 Alex Beal ***@***.***> wrote: Last I checked, the protobufs dep is used by more than just the proto rules: #726 (comment) <#726 (comment)> Maybe it could be factored out, but by the looks of that bazel query, it would be non-trivial. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#769>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAKQQF4MOQZEKPD7HWGU3ULP2P2IPANCNFSM4HYCHNCQ> .

[johnynek]

We have the deps to proto because the bazel worker protocol uses protobuf.

I had proposed a JSON API which wouldn’t bring such dependency problems since the protocol is so simple it would have been fine. But no one that I know of has worked on a PR.

I don’t know any way to keep workers and not have a protobuf dependency.

[ittaiz]

Fair enough. I’d guess that the protobuf dependency graph alone isn’t that big. The proto scala one was huge. @softprops, Maybe you can elaborate which deps hurt you?

…

On Sat, 15 Jun 2019 at 15:23 P. Oscar Boykin ***@***.***> wrote: We have the deps to proto because the bazel worker protocol uses protobuf. I had proposed a JSON API which wouldn’t bring such dependency problems since the protocol is so simple it would have been fine. But no one that I know of has worked on a PR. I don’t know any way to keep workers and not have a protobuf dependency. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#769>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAKQQF3666PKYSLF2FFG5RLP2TNNPANCNFSM4HYCHNCQ> .

[softprops]

A few weeks time had passed since we last tried an upgrade so I don't have the particular error hand. I was posting this in case others have ran into similar issues on their upgrade paths.

The protobuf error is stood out in particular for us because we knew that we already had a hard dependency on an older (2.4.1) version of protobuf on the jvm and protoc on the host we use for some internal codegen that we can't move away from but it was unclear why we were taking on a new version of this dependency when doing an upgrade the set of rules we only use for building scala_library and scala_tests.

I suspected that the extra protobuf dependency may have something to do with scala_rules bundling more than just core functionality so I was pondering if it wouldn't be worth splitting out the extra functionality and keeping scala rules limited to core scala build/test functionality. I totally get the utility of keeping them together. It eases the burden of testing them together and upgrading them together for those that use them together in their applications and libraries. I totally get that. For those that don't use them together in their applications it may cause bumps in the upgrade path.

It's worth mentioning for transparency we've been trying to get better about being able to separate our upgrades of bazel itself from rules_scala but its sometimes the case that upgrading bazel will have some implication on what version of rules_scala we can be using. My team in particular needs to pair both of those with working support of the bazel intellij plugin. So in our case, it's not just rules_scala we hit bumps on the upgrade path with. I'm thinking the fewer things any one of these things pull in would help minimize the number of things that can go wrong in an upgrade.

We've put down our last upgrade effort for now but when we come back to this, I'd be interested in helping more this issue forward.

[ittaiz]

First let me say that I completely hear you. We also juggle upgrading bazel/rules_scala/IJ and some others as well. I think Oscar's point about the worker protocol being bound to protobuf might be something you should take with the bazel team itself. If they move to a different protocol / support additional protocols then we can consider moving the worker (used for scala_library) to it. I'm not sure we'd like to be the only ones using a different experimental worker protocol.

…

On Sat, Jun 15, 2019 at 9:31 PM Doug Tangren ***@***.***> wrote: A few weeks time had passed since we last tried an upgrade so I don't have the particular error hand. I was posting this in case others have ran into similar issues on their upgrade paths. The protobuf error is stood out in particular for us because we knew that we already had a hard dependency on an older (2.4.1) version of protobuf on the jvm and protoc on the host we use for some internal codegen that we can't move away from but it was unclear why we were taking on a new version of this dependency when doing an upgrade the set of rules we only use for building scala_library and scala_tests. I suspected that the extra protobuf dependency may have something to do with scala_rules bundling more than just core functionality so I was pondering if it wouldn't be worth splitting out the extra functionality and keeping scala rules limited to core scala build/test functionality. I totally get the utility of keeping them together. It eases the burden of testing them together and upgrading them together for those that use them together in their applications and libraries. I totally get that. For those that don't use them together in their applications it may cause bumps in the upgrade path. It's worth mentioning for transparency we've been trying to get better about being able to separate our upgrades of bazel itself from rules_scala but its sometimes the case that upgrading bazel will have some implication on what version of rules_scala we can be using. My team in particular needs to pair both of those with working support of the bazel intellij plugin. So in our case, it's not just rules_scala we hit bumps on the upgrade path with. I'm thinking the fewer things any one of these things pull in would help minimize the number of things that can go wrong in an upgrade. We've put down our last upgrade effort for now but when we come back to this, I'd be interested in helping more this issue forward. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#769>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAKQQF3NRLINKPDGQ5KT4ILP2UYOJANCNFSM4HYCHNCQ> .