A few weeks ago I needed to parse configuration files in Scala for a Spark
project and decided to use
PureConfig. It is incredibly lean to
use, needing minimal boilerplate. I recommend you check it out (give also a look
at CaseClassy, which I haven’t had time
to test yet).
Everything seemed straightforward enough, and I got it working pretty quickly
(as in, it compiled properly). The surprise?
spark-submit failed with a
conflict with Shapeless (lacking a
witness). This is due to Spark 2.1.0 needing
Breeze, which in turn needs
Shapeless 2.0.0 (which is pretty old). Problem is, Spark’s required library
PureConfig from pulling the correct version. D’oh!
There is an easy fix, though, if you are using
sbt-assembly you can shade
dependencies, by adding something
like the following to your
assemblyShadeRules in assembly := Seq(ShadeRule.rename("shapeless.**" -> "new_shapeless.@1").inAll)