Decide on the role of `:` in indentation syntax

[odersky]

The meaning of : for significant indentation is more contentious than other aspects. We should come up with a crisp and intuitive definition where : is allowed.

One way I'd like to frame significant indentation in Scala 3, is that braces are optional, analogously to how semicolons are optional. But what does that mean? If we disregard : it means:

At all points where code in braces {...} is allowed, and some subordinate code is required, an indented code block is treated as if it was in braces.

The second condition is important, since otherwise any deviation from straight indentation would be significant, which would be dangerous and a strain on the eyes. But with that condition it's straightforward: If some subordinate code is required, either that code follows on the same line, or it is on a following line, in which case it should be indented. Braces are then optional, they are not needed to decide code structure. In the following I assume this part as given.

So that leaves the places where some code is not required but we still like to insert braces. There are two reasonable use cases for this:

1. Between the extends clause of a class, object, or similar and its (optional) template definitions in braces.
2. In a partial application such as

   xs.foreach:
     ...

   The motivation for this second case is to make uses of library-defined operations close to
   native syntax. If native syntax allows to drop braces, there should be a way for library-defined
   syntax to do the same.

Possible schemes

The current scheme relies on colons at end of lines that can be inserted in a large number of situations. There are several possible alternative approaches I can see:

1. Don't do it. Insist on braces for (1) and (2).

2. Split (1) and (2). Make indentation significant after class, object, etc headers without requiring a semicolon. This has the problem that it is not immediately clear whether we define an empty template or not. E.g. in

   object X extends Z
   
    // ...
    // ...
   
    object Y

   it is hard to see whether the second object is on the same level as the first or subordinate. But semantically it makes a big difference. So a system like that would be fragile. By contrast, a mandatory : would make it clear. Then the version above would be two objects on the same level and to get a subordinate member object Y you'd write instead:

   object X extends Z:
   
    // ...
    // ...
   
    object Y

   So I actually quite like the colon in this role.

3. Split (1) and (2) but require another keyword to start template definitions after class, object, etc headers. @eed3si9n suggested where. It's a possibility, but again I do like : at this point. It reads better IMO (and where might be a useful keyword to have elsewhere, e.g. in a future Scala with predicate refinement types).

4. Keep : (or whatever) to start templates but introduce a general "parens-killing" operator such as $ in Haskell with a mandatory RHS argument. If that occurred at the end of a line, braces would be optional according to our ground rules.

   I fear that a general operator like that would lead to code that was hard to read for non-experts. I personally find $ did a lot more harm than good to the readability of Haskell code.

5. Restrict the role of : to the two use cases above. I.e. allow a colon if

   • a template body is expected, or
   • in a partial application, after an identifier, closing bracket ] or closing parenthesis ).

   This way, we would avoid the confusing line noise that can happen if we allow : more freely.

My personal tendency would be to go for that last option.

[LPTK]

Regardless of whether we find a solution for (2), I would go for requiring braces for (1) — object, class, and trait definitions — which is what I proposed in the other thread.

But let me repeat and strengthen the rationale here:

• It provides a minimum amount of guidance while reading big nested definitions.

  I already find it really difficult to read sources like this or this, where there are several levels of nested objects/classes with huge bodies. It's really challenging to see what begins and ends where — just imagine what it would look like without braces. Currently, using an IDE helps, as I can place the cursor on an opening or closing brace and see the whole corresponding scope highlighted/delineated. That won't be possible without braces.

  And as others have pointed out, this kind of code does come up in realistic situations (my examples are from the scala/scala repo after all). If we make braces around classes/objects optional, we'll encourage people not to use them, and we'll invariably end up with unreadable messes when files naturally grow in size over time.

• It makes sense conceptually, if we consider that curly braces are used specifically to denote object scopes.

• I would also argue that object scopes should be explicitly delineated because they are much more significant than expression scopes. Indeed, in an object scope, you get a bunch of implicitly imported definitions, you get a new meaning for this, etc.

[odersky]

@LPTK I believe that end markers are actually a far superior way to delineate class scopes. And having both braces and end markers would look weird.

[JanBessai]

Not sure if this is the correct thread, but:
How does indentation sensitivity mix with triple quoted multi-line strings? Does the end-quote have to be indented? If so: how do you write down a line break + spaces before the end quote? Indent further?

[sjrd]

At the risk of having to my own devil's advocate in a week or so ... what about using with instead of : for the two use cases above.

For templates:

class Foo extends Bar with SomeTrait with
  def x: Int = 42

The with there does not shock me. It is even quite easy to interpret it as "braces as optional" if we also allow

class Foo extends Bar with SomeTrait with {
  def x: Int = 42
}

That syntax makes sense to me, as the contents of the block are added as members of Foo just as much as the members of SomeTrait. I don't know, it kind of makes sense to me.

For that, the lexer would introduce an indent if with is at the end of the line, the next line is indented and starts by anything but an identifier.

I would actually prefer to simply not have anything to open a template, though. However this might require the parser to feed into the lexer to implement, which is not ideal. Or maybe we can make it work by simply counting opening and closing brackets? That would be nice.

For method calls:

xs.foreach with
  x =>
    println(x)

val y = xs.foldLeft(0) with
  (prev, x) =>
    prev + x

val z = optInt.fold with
  println("nope")
with
  x => println(x)

This syntax had actually been proposed in the original "new implicits" proposals, for implicit parameter lists. So it must have appealed at some point. Except here it is used to pass normal arguments, when they are blocks (by-name params or lambdas).

The advantage over : is that, well, it's not :. There have been several concerns about the multiple problems that : exposes, not the least being the overloading wrt. type ascriptions.

with also suffers from overloading in this case, but perhaps it is less annoying than : because it would only have two meanings that are very well visually separated: with in a class header is composition; with in an expression is a block argument.

I don't really like what I'm proposing, but I dislike it a lot less than :.

[jducoeur]

I don't really like what I'm proposing, but I dislike it a lot less than :.

Strong agreement. I think : is terrible -- not only is it badly overloaded, it's simply too visually subtle. I'm concerned that it will lead to bugs due to people just missing it. I dislike this whole approach (IMO it crosses the line into designing a different language, not enhancing Scala, and enormously increases the risk of splitting the community) but probably this aspect most of all...

[eed3si9n]

In #7083 I wrote:

Since colon means type annotation (ascription) x: A already, reusing this to mean "begin bill of material" or "begin block" seems odd to me too.

F# uses =, and Haskell uses where:

module Main where  
  import A  
  import B  
  main = A.f >> B.f

(2) In a partial application such as xs.foreach:

I think with is an improvement over :, but it does suffer from overloading. Could we borrow <| from F# here?

xs.foreach <| x =>
  println(x)

ys.foreach <| y => println(y)

val y = xs.foldLeft(0) <| (prev, x) =>
  prev + x

If you squint, it looks like begin {. Note here that the indentation doesn't start until \n so you can write x => on the same line.

pattern match

Pattern matching is an odd one because <| is the opposite direction.

val kind = ch match
  case ' '  => "space"
  case '\t' => "tab"
  case _    => s"'$ch'-character"

You basically want |> but that might be too confusing:

val kind = ch |>
  case ' '  => "space"
  case '\t' => "tab"
  case _    => s"'$ch'-character"

I think we should just allow match to be a special indentation introducer.

(1) template definitions

What's interesting about template is that it is both a list of members and it's also the body of the constructor. So in that sense it might make sense to keep it the same syntax as block introducer.

class Contact(name: String) extends Bar with SomeTrait <|
  def x: Int = 42

  DB.append(name)

object Contact <|
  def apply: Contact = new Contact("")

I understand that = is not exactly right, but visually it looks less intimidating I think

class Contact(name: String) extends Bar with SomeTrait =
  def x: Int = 42

  DB.append(name)

object Contact =
  def apply: Contact = new Contact("")

I don't really like what I'm proposing, but I dislike it a lot less than :.

Ditto.

[lihaoyi]

I propose re-using existing keywords wherever possible. The whole point of this exercise is to make the syntax lightweight: having long keywords like where defeats the purpose entirely. if necessary, we could commandeer do as a short-enough keyword to delimit blocks in the case of ambiguity. I think : is common enough that overloading it would be confusing, and it really isn't that much shorter than do anyway.

Re-using existing keywords can get us surprisingly far. Consider Scalite:

package scalite.tutorial                                    package scalite.tutorial

class Point(xc: Int, yc: Int)                               class Point(xc: Int, yc: Int) {
    var x: Int = xc                                           var x: Int = xc
    var y: Int = yc                                           var y: Int = yc
    def move(dx: Int, dy: Int) =                              def move(dx: Int, dy: Int) = {
        x = x + dx                                              x = x + dx
        y = y + dy                                              y = y + dy
                                                              }
    override def toString() =                                 override def toString() = {
        "(" + x + ", " + y + ")"                                "(" + x + ", " + y + ")"
                                                              }
                                                            }
object Run                                                  object Run {
    def apply() =                                             def apply() = {
        val pt = new Point(1, 2)                                val pt = new Point(1, 2)
        println(pt)                                             println(pt)
        pt.move(10, 10)                                         pt.move(10, 10)
        pt.x                                                    pt.x
                                                              }
                                                            }

var x = 0                                                   var x = 0
for(i <- 0 until 10)                                        for(i <- 0 until 10) {
    val j = i * 2                                             val j = i * 2
    val k = j + 1                                             val k = j + 1
    x += k                                                    x += k
                                                            }
val list =                                                  val list = {
    for(i <- 0 to x) yield                                    for(i <- 0 to x) yield {
        val j = i + 1                                           val j = i + 1
        i * j                                                   i * j
                                                              }
                                                            }
list.max                                                    list.max
// 10100                                                    // 10100

val all = for                                               val all = for {
    x <- 0 to 10                                              x <- 0 to 10
    y <- 0 to 10                                              y <- 0 to 10
    if x + y == 10                                            if x + y == 10
yield                                                       } yield {
    val z = x * y                                             val z = x * y
    z                                                         z
                                                            }
all.max                                                     all.max
// 25                                                       // 25

I think this looks superior to using any delimiter. Note that the above already works in Scala 2.11/12/13, and has worked for 5 years now.

We have to use the same whitespace rules for both def/var/vals and class/trait/object. Having mixed rules for what things are "whitespace compatible" and what things aren't is a mess, and would look terribly ugly, especially in Scala where (unlike Java) having nested inner class/objects at the same level as your def/var/vals is common, and having top-level def/var/vals at the same level as your top-level class/objects is going to become common as well. If we want classes/objects to be "special", that ship sailed long ago.

One subtlety that we have to take care of is providing for higher-order methods. Code like:

val foo = bar
  .map{x => 
    val y = x + 1
    y + 1
  }
  .foreach{ x =>
    val y = x + 1
    println(y)
  }

is extremely common with Scala's method-chaining conventions, and I'd want to be able to provide a whitespace-compatible syntax:

val foo = bar
  .map x => 
    val y = x + 1
    y + 1
  .foreach x =>
    val y = x + 1
    println(y)

In Scalite I commandeered the do keyword for this purpose:

val xs = 0 until 10                                         val xs = 0 until 10
val ys = xs.map do                                          val ys = xs.map{
    x => x + 1                                                x => x + 1
                                                            }
ys.sum                                                      ys.sum
// 55                                                       // 55

val zs = xs.map do                                          val zs = xs.map{
case 1 => 1                                                   case 1 => 1
case 2 => 2                                                   case 2 => 2
case x if x % 2 == 0 => x + 1                                 case x if x % 2 == 0 => x + 1
case x if x % 2 != 0 => x - 1                                 case x if x % 2 != 0 => x - 1
                                                            }
zs.sum                                                      zs.sum
// 45                                                       // 45

val ws = xs.map do x =>                                     val ws = xs.map { x =>
    val x1 = x + 1                                            val x1 = x + 1
    x1 * x1                                                   x1 * x1
                                                            }
ws.sum                                                      ws.sum
// 385                                                      // 385

Since we seem to be deprecating do-while loops as I suggested 5 years ago, the do keyword is freed up. As a short, 2-character, now entirely unconflicted keyword, we should make sure we use it as effectively as possible.

For a longer example using what I propose, please take a look at this self-contained JSON parser:

• https://github.com/lihaoyi/Scalite/blob/master/example/src/test/scalite/tutorial/Js.scalite

I think it would be a good starting point to compare different syntaxes, being meaty enough to really give you a feel of things where trivial 10-line examples do not. The short examples are also illustrative:

• https://github.com/lihaoyi/scalite

[odersky]

I want to comment on sub-part (2), i.e. what to use for starting an indented argument. I believe that no single keyword would work well in that role.

with was borderline acceptable as an implicit argument, since it evokes pairing with a context. But for plain arguments it looks wrong at least as often as it looks OK. For instance:

math.logarithm with 
  val x = f(y)
  x * x

I believe with would only work well if the real function application is to something else and the part following with is in some way an accessory to that. E.g. in xs.map with f it works since the mapped argument is xs and f is an accessory. But the case of logarithm above shows that we cannot generalize that to all applications.

do evokes side effects. It is used now in that role specifically in while or for loops. E.g.:

for 
  x <- xs
  y <- ys
do
  println(x + y)

I believe for pure functions, do is out of place. E.g.:

transpose do
  val a: Matrix = ...
  val b: Matrix = ...
  a * b

In fact, the pattern of function application is so general and multi-faceted that no single keyword can do it justice. That's why Dijkstra uses infix point (well, that's taken already in Scala!) and Haskell uses $. But colon does work brilliantly in this role! Evidence #1 is Python, where it is perceived to be very natural. Yes, I know Python uses it everywhere instead of just for this purpose, but still... Evidence #2 is common language. It's very natural to write something like:

  To make a cake:
    preheat oven,
    mix flours, eggs and milk,
    bake for one hour.

The : specifically introduces a list of statements that's subordinate to a prefix clause. That's completely grammatical. So, I believe strongly that : is the best operator for this.

As to possible ambiguity with type ascription: I really don't think that's a problem. : at the end of line and : used infix are visually quite distinct. The only caveat is that we should not let one follow the other. I.e.:

  def f(): T:
     return foo

would be awkward. But ever since procedure syntax was dropped, Scala does not have syntax where this pattern could occur.

The only possible ambiguity is in a type ascription of an expression spanning multiple lines like this:

  someLongExpression:
    someLongType

But I have not seen code like this in the wild, and in fact our code base including all tools, all tests, and community build does not have a single instance where this pattern occurs. Why does it not occur?
I guess if you have long expressions and types to combine you realize that you are probably better off factoring stuff into a val:

  val someId: someLongType = 
    someLongExpression
  someId

And, if you really need to write a multi-line ascription, you can aways do:

  someLongExpression
    : someLongType

which in fact reads much better. So, in summary, any ambiguity would be extremely rare, and is easily avoided. The other evidence why ambiguities are not a concern is again Python. Python does use : for both roles and the Python community is not known to be sloppy with syntax. Interestingly Python does not use : to indicate a function return type, since that would let them run into precisely the awkwardness I referred to earlier. It uses -> instead. But since Scala uses = instead of : to start a function body it does not run into that problem.

One downside of : is that it only works at the end of lines. So:

  xs.map: x =>
    val y = f(x)
    g(y)

does not work. You have to format it instead as:

  xs.map: 
    x =>
    val y = f(x)
    g(y)

I think this is not so bad. In real code the { x => part is often quite far to the right because the expression preceding it is long. This makes it hard to see the bound name. The vertical syntax makes it much clearer what is bound.

[lihaoyi]

I think this is not so bad. In real code the { x => part is often quite far to the right because the expression preceding it is long. This makes it hard to see the bound name. The vertical syntax makes it much clearer what is defined.

To me this formatting is he deal breaker, much more than the ambiguity around type ascriptions. I work with a lot of real code formatted exactly as you describe, and it reads excellently. If the LHS is long, the .map goes on a new line.

I have seen no code at all formatted similar to how you propose, and subjectively it looks awful. If it really looked better, people would already be formatting their lambdas like that right now, and they’re not.

[odersky]

@lihaoyi

I have seen no code at all formatted similar to how you propose, and subjectively it looks awful. If it really looked better, people would already be formatting their lambdas like that right now, and they’re not.

Fair point.

Here's a crazy idea for this pattern: use case! Examples:

xs.map case x =>
  val y = f(x)
  g(y)

xs.collect case Some(n) => n

xs.foreach case i => println(s"next: $i")

Points in favor:

• case is semantically a correct option here.
• we can combine this with a pattern match.
• indentation works out of the box, but now it's prompted by the => at the EOL.

To make this work we'd have to add one production to Expr1. It's the last line below:

Expr1 ::= ...
             |  Expr2 ‘match’ ‘{’ CaseClauses ‘}’
             |  Expr2 ‘case’ Pattern [Guard] ‘=>’ Expr

WDYT?

[odersky]

[Aside: You may have noted that I "ate my own dog food" in the comments above: every single indented section was introduced with :. Readers should judge for themselves whether this is natural or not.]

[lihaoyi]

case looks ok-ish to me, but that seems to introduce an even worse ambiguity: that between partial functions and total functions! This is something we had already made efforts to disambiguate in Dotty (e.g. requiring case for partial functions for for-comprehensions), so that case always means partiality, and partiality always means case. Requiring case just to make indentation work properly is definitely a step backwards

Honestly, I think we should just use do. The English meaning is almost irrelevant: with, class, object, for, etc. in Scala already mean vastly different things in English and Scala, and that's been mostly OK. Having do mean something specific in Scala isn't going to be the end of the world, especially since it's going to be entirely unambiguous and consistent: do will have no other meanings once do-while loops are out. I think it would definitely be preferable to overloading with or case or some other keyword which have very specific, existing Scala meanings.

I appreciate the desire to use :, but function-literals-with-arguments is a real sticking point. Python can get by with (1) not having multiline lambdas (2) a different -> foo syntax for annotating return types and (3) a dedicated syntax for context managers. Scala has none of these things, and passing multi-line n-arg function literals to higher order functions is the order of the day. We have to make sure it looks pretty and first class

[odersky]

I have strong objections against do. do universally means imperative side effect, in natural language as well as in all programming languages I know. (do comprehensions in Haskell model side effects via monads). So I believe we cannot simply change its meaning, in particular in a language like Scala which is predominantly functional but still allows side effects.

case is often used with partial functions but not always. A counter example is:

case class Point(x: Double, y: Double)

val points: List[Point] = ...
points.map {
  case Point(x, y) => ...
}

So, case is also used for destructuring in total functions. A simple variable pattern is a special case of that (and it's obvious at a glance that that's what it is)