Why you can’t always flatMap to Option

4 minutes read

A little while ago, something that had always worked for me in Scala suddenly didn’t work. Here’s what happened.

In Scala, many functions return Option[T], indicating failure with None and success with Some(result). For example, this parsing function returns None when the string can’t be parsed:

def parse(s: String): Option[Int] = scala.util.Try { s.toInt }.toOption

If I’m parsing several strings, and want to discard the ones that don’t parse, then this kind of function works well with flatMap:

scala> List("1","2","nope","2.71828","3").flatMap(parse)
res0: List[Int] = List(1, 2, 3)

Here Option[T] acts like a List[T] that contains zero or one elements.

One day, for some reason I can’t remember, I tried putting parse into a variable:

scala> val parse2 = parse
<console>:12: error: missing argument list for method parse
Unapplied methods are only converted to functions when a function type is expected.
You can make this conversion explicit by writing `parse _` or `parse(_)` instead of `parse`.
       val parse2 = parse
                    ^

I didn’t really understand the error, but I tried parse _ as it suggested. (Everything is the same as parse when using parse(_).)

scala> val parse2 = parse _
parse2: String => Option[Int] = <function1>

Okay. But then this:

scala> List("1","2","nope","2.71828","3").flatMap(parse2)
<console>:10: error: type mismatch;
 found   : String => Option[Int]
 required: java.lang.String => scala.collection.GenTraversableOnce[?]
              List("1","2","nope","2.71828","3").flatMap(parse2)
                                                         ^

Annoyingly, the error message makes sense. The found type is confirmed just above. The required type matches what the standard library documentation says. And Option[T] doesn’t extend the trait GenTraversableOnce[T]. So yeah, this code doesn’t type-check.

But then why does it work with .flatMap(parse)?

I tried some random changes, as one does. Does it want another underscore? No:

scala> List("1","2","nope","2.71828","3").flatMap(parse2 _)
<console>:10: error: type mismatch;
 found   : () => String => Option[Int]
 required: java.lang.String => scala.collection.GenTraversableOnce[?]
              List("1","2","nope","2.71828","3").flatMap(parse2 _)

Is it that parse works but parse _ doesn’t? No:

scala> List("1","2","nope","2.71828","3").flatMap(parse _)
res3: List[Int] = List(1, 2, 3)

What about hiding parse2 in an anonymous function? Yes:

scala> List("1","2","nope","2.71828","3").flatMap(x => parse2(x))
res4: List[Int] = List(1, 2, 3)

(The reporter of SI-9606 tried a few more things. Their conclusion: “Once again we are left with the message that ‘Scalac occasionally does weird shit that only the high priests understand’.”)

To save you the time digging through the language spec, here’s what’s going on.

How it works when we pass an anonymous function

Here’s that last example again:

scala> List("1","2","nope","2.71828","3").flatMap(x => parse2(x))
res4: List[Int] = List(1, 2, 3)

When the compiler type-checks the argument to flatMap, it faces this situation:

Expression:     x => parse2(x)

Expected type:  String => GenTraversableOnce[B]
Actual type:    S => T

The expected type comes from the signature of flatMap. (B is the type parameter of flatMap, representing the type of the elements of the resulting list.) The so-called “actual type” here is what the compiler assumes for anonymous functions without type annotations. Comparing these types, the compiler concludes that S = String and T = GenTraversableOnce[B].

(This isn’t exactly how the language spec describes this situation, but it works out the same.)

So far so good. To finish type-checking x => parse2(x), the compiler type-checks the body of the function:

Expression:     parse2(x)

Expected type:  GenTraversableOnce[B]
Actual type:    Option[Int]

The expected type comes from previous step. The actual type comes from the type of parse2; namely String => Option[Int].

But Option doesn’t extend GenTraversableOnce, so these types aren’t compatible. To recover, the compiler looks for an implicit conversion to save the day. It finds this one in Option’s companion object:

implicit def option2Iterable[A](xo: Option[A]): Iterable[A]

Using this implicit turns the situation into:

Expression:     option2Iterable(parse2(x))

Expected type:  GenTraversableOnce[B]
Actual type:    Iterable[Int]

Since Iterable[Int] extends GenTraversableOnce[Int], these types are compatible (taking B = Int), and the compiler rests.

How it doesn’t work if we pass a named function object

Here’s the failing case again:

scala> val parse2 = parse _
parse2: String => Option[Int] = <function1>

scala> List("1","2","nope","2.71828","3").flatMap(parse2)
<console>:10: error: type mismatch;
 found   : String => Option[Int]
 required: java.lang.String => scala.collection.GenTraversableOnce[?]
              List("1","2","nope","2.71828","3").flatMap(parse2)

The difference between passing a variable and passing an anonymous function is that parse2 is known to have a more precise type. This time, the compiler faces this situation:

Expression:     parse2

Expected type:  String => GenTraversableOnce[B]
Actual type:    String => Option[Int]

Again, Option doesn’t extend GenTraversableOnce, so these types aren’t compatible. This time there’s no suitable implicit: it would need to be a conversion from, say, String=>Option[Int] to String=>Iterable[Int], but there isn’t one.

This diagnosis is confirmed by trying it with a suitable implicit:

scala> implicit def func2Option2Func2Iterable[A,B](f: A=>Option[B]): A=>Iterable[B] = x => f(x)
func2Option2Func2Iterable: [A, B](f: A => Option[B])A => Iterable[B]

scala> List("1","2","nope","2.71828","3").flatMap(parse2)
res5: List[Int] = List(1, 2, 3)

Exercise: How does the body of func2Option2Func2Iterable type-check?

How it works if we pass a method

Finally, here’s the usual case, where I pass a method previously defined by def:

scala> List("1","2","nope","2.71828","3").flatMap(parse)
res0: List[Int] = List(1, 2, 3)

This looks a lot like passing the function parse2 (which didn’t work), but methods and functions are not actually the same kind of thing. When a function is expected and a method is found, the method is converted to a function: parse2 is treated like x => parse2(x), which is an anonymous function, so it works as described above.

(Functions are first-class objects in Scala, but methods aren’t; you could say that the language is functional but not methodical.)

tl;dr

flatMap-to-Option relies on a delicate interaction between implicits and local type inference. It’s delicate enough to fail if you so much as replace an expression with a variable whose value is that same expression:

List("1","2","nope","2.71828","3").flatMap(parse _)          // OK!

val parse2 = parse _
List("1","2","nope","2.71828","3").flatMap(parse2)           // NOPE

It’s also delicate enough to fail if you replace an anonymous function whose body is just a call to another function with a direct call to that function:

List("1","2","nope","2.71828","3").flatMap(x => parse2(x))   // OK!

List("1","2","nope","2.71828","3").flatMap(parse2)           // NOPE

¯\_(ツ)_/¯

Updated: