Apache Spark & Scala

Using Enumerations with Apache Spark

Published on June 1, 2022 by Viliam Simko · Updated September 12, 2024

I'm a senior freelance developer specializing in Scala and Apache Spark. I primarily work with companies in the fintech domain and help them tackle their big data challenges.

In this blog post, I want to address a surprisingly small but important design decision that comes up regularly when working with Spark's Dataset API: how should we represent enumerated values in Scala?

At first glance, this seems trivial. A status can simply be a String:

case class User(
  id: Long,
  name: String,
  status: String
)

But once the same status values are used throughout a larger data pipeline, strings quickly become a source of bugs. Should "ACTIVE" be written as "ACTIVE", "Active", or "active"? What happens when somebody renames a status? And what happens when a new status is introduced but some transformation forgets to handle it?

Scala gives us several ways to solve this problem. We can use the built-in Enumeration, sealed trait hierarchies, libraries such as Enumeratum, or Java enums.

I have tried several of these approaches in real Spark projects. Interestingly, my eventual conclusion was not the most sophisticated Scala solution.

For Spark projects, plain Java enums turned out to be the most pragmatic choice for me.

Why Strings Fail in Distributed Pipelines

In large-scale data engineering with Apache Spark, developers frequently represent categorical status flags and domain codes as raw String values.

case class User(
  id: Long,
  name: String,
  status: String
)

dataset.filter(_.status == "ACTIVE")

The problem is that the compiler has no idea that "ACTIVE" is a valid status while "ACTVE" is a typo.

dataset.filter(_.status == "ACTVE")

In a distributed pipeline, this can be particularly unpleasant because the mistake may only become visible after the Spark job has processed a significant amount of data.

What we really want is something more like this:

dataset.filter(_.status == UserStatus.ACTIVE)

Now the compiler knows what UserStatus is. The domain concept is explicit in the code instead of being hidden inside arbitrary strings.

Scala Enumeration

Scala has a built-in enumeration mechanism:

object UserStatus extends Enumeration {
  val ACTIVE, SUSPENDED, PENDING = Value
}

case class User(
  id: Long,
  name: String,
  status: UserStatus.Value
)

This is already considerably better than using strings. The compiler can distinguish the enumeration value from an arbitrary string, and Spark's Scala reflection machinery has explicit support for Scala Enumeration values.

However, the Scala Enumeration API is relatively old and does not integrate particularly elegantly with the rest of the Scala type system. Pattern matching and exhaustive handling are also not as pleasant as with sealed hierarchies.

Sealed Traits and Case Objects

A more idiomatic Scala approach is to model the states as a sealed hierarchy:

sealed trait UserStatus

object UserStatus {
  case object Active extends UserStatus
  case object Suspended extends UserStatus
  case object Pending extends UserStatus
}

case class User(
  id: Long,
  name: String,
  status: UserStatus
)

And pattern matching becomes particularly attractive:

status match {
  case UserStatus.Active =>
    // ...
  case UserStatus.Suspended =>
    // ...
  case UserStatus.Pending =>
    // ...
}

Because the hierarchy is sealed, the Scala compiler can reason about all known variants and can warn when a pattern match is not exhaustive.

This is a very nice solution from a pure Scala perspective. But Spark introduces another consideration.

Spark Needs an Encoder

A Scala type is not automatically a Spark SQL type. Spark's Dataset[T] API relies on an Encoder[T] to convert domain objects into Spark's internal representation.

For ordinary case classes, Spark can usually derive the required encoder automatically. For custom algebraic data types, however, things become more complicated.

A sealed hierarchy is an excellent Scala domain model, but it is not simply a built-in Spark SQL enum representation. Depending on the exact representation you want in Spark, you may need additional encoder or conversion code.

You can instead keep the Scala ADT at the application boundary and convert it to a Spark-friendly representation such as a string or another supported type. But that means introducing another conversion layer.

This is where the theoretical elegance of a solution starts competing with practical concerns.

Enumeratum

If you want stronger enumeration support in Scala, one popular option is Enumeratum.

Enumeratum provides type-safe enumerations and useful functionality around enumeration values, parsing, serialization and integrations.

import enumeratum._

sealed trait UserStatus extends EnumEntry

object UserStatus extends Enum[UserStatus] {
  case object Active extends UserStatus
  case object Suspended extends UserStatus
  case object Pending extends UserStatus

  val values = findValues
}

This gives us a much richer enumeration abstraction than the old Scala Enumeration.

Enumeratum is particularly useful when the enumeration itself is an important part of the application's domain model.

But again, Spark changes the equation. Enumeratum does not automatically become a native Spark SQL type simply because it is a type-safe Scala enumeration. When an enum is used extensively inside Spark Dataset transformations, joins and encoders, we also need to consider how that representation crosses Spark's encoder boundary.

Depending on the exact Enumeratum type and Spark integration you choose, this may mean explicit conversions or additional encoder/serialization integration.

Java Enums

This brings us to the surprisingly pragmatic option: the good old Java enum.

public enum UserStatus {
    ACTIVE,
    SUSPENDED,
    PENDING
}

It can then be used directly from Scala:

case class User(
  id: Long,
  name: String,
  status: UserStatus
)

dataset.filter(_.status == UserStatus.ACTIVE)

From a Scala developer's perspective, this may initially feel slightly less elegant than a sealed trait hierarchy.

But there is an important practical advantage: Java enums fit Spark's existing encoder model directly.

Spark's encoder machinery explicitly recognizes Java enum types and can create the corresponding Java-enum encoder. This means we can use a normal Java enum as part of a typed Dataset without introducing a separate enum-specific encoding abstraction.

The Encoder Trade-Off

This is the part that ultimately changed my decision.

In a pure Scala application, I would be much more inclined to use a sealed hierarchy or Enumeratum. But in a Spark application, the type does not live only in the domain model. It has to cross Spark's Dataset and encoder boundary repeatedly.

A useful way to think about the alternatives is:

Scala ADT
   ↓
custom / explicit representation
   ↓
Spark SQL


Enumeratum
   ↓
Spark encoder / serialization integration
   ↓
Spark SQL


Java enum
   ↓
Spark's existing enum encoder support
   ↓
Spark SQL

With a Java enum, the representation is already something Spark's encoder machinery knows how to handle. With a custom Scala hierarchy or third-party enumeration abstraction, the integration itself can become part of the application's codebase and maintenance burden.

This is not primarily a runtime-performance argument. The advantage is primarily about integration complexity, encoder compatibility and maintainability.

Why I Eventually Preferred Java Enums

After experimenting with different approaches, my practical conclusion was that Java enums were the best fit for the Spark projects I was working on.

Not because Java enums are somehow more powerful than Enumeratum or a Scala sealed hierarchy. They aren't. The advantage was simply integration.

A Java enum:

is strongly typed,
has a straightforward representation,
is well understood by developers coming from both Scala and Java,
fits naturally into Spark's existing encoder machinery,
does not require an additional enumeration library,
avoids custom encoder infrastructure for a simple use case, and
works well when the enum is primarily a finite set of values rather than a sophisticated Scala domain abstraction.

This is one of those cases where the most elegant solution in isolation is not necessarily the best solution for the entire system.

Type Safety Still Matters

Using Java enums does not mean giving up the advantages of type safety.

dataset.filter(_.status == "ACTIVE")

is replaced by:

dataset.filter(_.status == UserStatus.ACTIVE)

The second version makes the domain explicit. The compiler knows that status is a UserStatus, and it can prevent accidental comparisons with arbitrary strings.

This becomes particularly valuable when the same status appears in many transformations:

dataset
  .filter(_.status == UserStatus.ACTIVE)
  .map(...)
  .join(...)
  .filter(...)

The domain concept remains visible throughout the pipeline.

What About Serialization?

There is another important consideration when using enums in data pipelines: how the value is represented outside the application.

Your business domain may use UserStatus.ACTIVE, while an external system may represent the same value as "ACTIVE" or another external code.

These are different concerns.

The enum gives us a type-safe representation inside the application. At system boundaries, we may still need explicit conversion to and from the representation used by Kafka, JSON, databases, files, or external APIs.

The important point is that we should not let the external representation leak unnecessarily into the rest of the application.

A Practical Decision Matrix

There is no universal winner. The choice depends on what the enumeration needs to do and how deeply the type is integrated with Spark.

Approach	Scala domain modelling	Spark encoder friction	Extra dependency
`String`	Poor	None	No
Scala `Enumeration`	Moderate	Low	No
Sealed trait / case objects	Excellent	Higher	No
Enumeratum	Excellent	Potentially higher	Yes
Java `enum`	Good	Low	No

The important lesson is that Spark integration should be part of the decision.

If I were building a pure Scala domain model with no Spark dependency, I might choose differently. But when the same type has to travel through a large Spark Dataset pipeline, the encoder story becomes an important part of the design.

Key Learnings

Avoid raw strings for finite domain values.
Type safety is particularly valuable in long-running distributed pipelines.
Scala sealed hierarchies provide excellent domain modelling and exhaustive pattern matching.
Enumeratum is a powerful option when you need richer Scala enumeration functionality.
Spark's encoder support should be considered before choosing a domain representation.
Java enums are a surprisingly practical choice for Spark Dataset applications.
The simplest solution that integrates cleanly with the whole stack is often better than the most sophisticated abstraction.

Final Thoughts

When I first started looking at this problem, I was naturally attracted to the most type-safe and idiomatic Scala solutions.

Over time, however, real projects taught me a slightly different lesson.

A data engineering codebase is not just Scala code. It is Scala, Spark SQL, encoders, serialization, storage formats, Kafka, databases, build tooling, and often Java libraries and infrastructure.

A solution that looks perfect from the Scala language perspective can introduce unnecessary friction somewhere else.

For my Spark projects, plain Java enums turned out to be a pragmatic sweet spot: they provide the type safety I want while fitting naturally into Spark's existing encoder machinery.

If the enum is primarily a domain concept inside a pure Scala application, use the Scala abstraction that best fits your domain. If the enum is going to live deeply inside Spark Datasets, don't underestimate the value of boring, well-supported Java enums.

Sometimes the most pragmatic solution is the most maintainable one.

← Back to all articles