Apache Spark & Scala

Working with Nested JSON Data in Apache Spark

Published on December 18, 2022 by Viliam Simko · Updated September 12, 2024

I'm a senior freelance developer specializing in Scala and Apache Spark. I primarily work with companies in the fintech domain and help them tackle their big data challenges.

When working with Spark, we often encounter data that is naturally nested. A customer may have an address, an order may contain a list of items, or a business event may contain a complex metadata structure.

Spark handles these structures naturally through its SQL data types such as struct, array and map. The problem usually appears at the system boundary: we eventually need to send that nested structure to a database, Kafka, an API, or another system that expects JSON.

In this article, I want to look at a simple and practical approach to this problem: keep the data in Spark's native representation for as long as possible, and serialize it to JSON only when we actually need to cross a system boundary.

The Problem: Nested Data in Relational Systems

Consider a simple domain model:

case class Address(
  street: String,
  city: String,
  country: String
)

case class Customer(
  id: Long,
  name: String,
  address: Address
)

In Scala, this is a perfectly natural representation. Spark can also represent the address field as a nested structure.

But suppose that our target database has a table like this:

CREATE TABLE customers (
  id BIGINT,
  name TEXT,
  address_json TEXT
)

We now have a mismatch between the two representations:

Scala / Spark

Customer
  ├── id
  ├── name
  └── address
       ├── street
       ├── city
       └── country


Relational database

customers
  ├── id
  ├── name
  └── address_json

We need to turn the nested Spark structure into a JSON string before writing it to the database.

Spark's Native JSON Functions

Spark already provides functions for exactly this purpose. The two functions that are particularly useful here are to_json and from_json.

Conceptually, they form a simple pair:

Spark structure
      ↓
   to_json
      ↓
  JSON string


  JSON string
      ↓
  from_json
      ↓
Spark structure

The important detail is that these are Spark SQL functions. They operate on Spark columns rather than requiring us to convert every row into a Scala object and serialize that object ourselves.

Serializing Nested Structures with `to_json`

Let's start with the simple case. Suppose we have a Dataset of customers:

import spark.implicits._

val customers: Dataset[Customer] = ...

We can serialize the nested address structure into a JSON string using to_json:

import org.apache.spark.sql.functions._

val result =
  customers
    .withColumn("address_json", to_json($"address"))

The resulting data contains both the original nested structure and its JSON representation:

+----+------+----------------------------------------+
|id  |name  |address_json                            |
+----+------+----------------------------------------+
|1   |Alice |{"street":"Main Street","city":"Prague"} |
+----+------+----------------------------------------+

If we no longer need the original nested column, we can remove it:

val result =
  customers
    .withColumn("address_json", to_json($"address"))
    .drop("address")

At this point, address_json is simply a Spark StringType column. That makes it easy to pass it to a downstream system that expects a string.

Why Not Serialize Inside `map`?

A tempting alternative is to leave Spark SQL and serialize the Scala object manually:

dataset.map { customer =>
  objectMapper.writeValueAsString(customer)
}

There is nothing inherently wrong with using Jackson or another JSON library. The important question is which layer should perform the serialization.

If the data is already represented as Spark SQL data, using to_json keeps the operation inside Spark's SQL expression model.

With map plus a general-purpose JSON library, we instead move the operation into the JVM closure executed for each record. That introduces an unnecessary transition from Spark's representation back to application objects just to serialize them again.

In other words, I prefer:

Spark SQL structure
        ↓
      to_json
        ↓
    JSON string

over:

Spark SQL structure
        ↓
     Scala object
        ↓
   Jackson / Circe
        ↓
    JSON string

This is not a rule that general-purpose JSON libraries should never be used in Spark. They are useful when we need functionality that belongs to application-level serialization rather than Spark SQL serialization.

Reading JSON Back with `from_json`

The opposite operation is useful when JSON arrives from an external system and we want to turn it back into a Spark SQL structure.

For example, imagine a column containing:

{"street":"Main Street","city":"Prague","country":"CZ"}

We can parse it using from_json and an explicit Spark schema:

import org.apache.spark.sql.types._

val addressSchema = new StructType()
  .add("street", StringType)
  .add("city", StringType)
  .add("country", StringType)

val parsed =
  df.withColumn(
    "address",
    from_json($"address_json", addressSchema)
  )

The result is a Spark struct column containing the individual fields.

This gives us a useful round trip:

struct
  ↓
to_json
  ↓
JSON
  ↓
from_json
  ↓
struct

One important difference is that from_json needs to know the expected schema. In production pipelines, that schema is part of the contract between the producer and consumer.

Writing Nested JSON to a Relational Database

This pattern is particularly useful when Spark is used as a data processing layer in front of a relational database.

For example, suppose PostgreSQL contains a table with a JSON-capable column:

CREATE TABLE customers (
  id BIGINT,
  name TEXT,
  address_json JSONB
)

On the Spark side, we can prepare the data first:

val result =
  customers
    .withColumn("address_json", to_json($"address"))
    .drop("address")

We can then write the resulting DataFrame using Spark's JDBC support:

result.write
  .format("jdbc")
  .option("url", jdbcUrl)
  .option("dbtable", "customers")
  .option("user", dbUser)
  .option("password", dbPassword)
  .mode("append")
  .save()

The important boundary here is that Spark sees address_json as a StringType column. How the target database stores that value — for example as TEXT or JSONB — is a separate database-level concern and can depend on the JDBC driver and target database.

In other words, to_json solves the Spark-side serialization problem. It does not replace the database's own type system.

Serialize as Late as Possible

One of the most useful lessons I learned from working with Spark pipelines is to avoid turning structured data into JSON too early.

Consider a pipeline like this:

read
  ↓
to_json
  ↓
filter
  ↓
join
  ↓
aggregation
  ↓
write

Once the nested structure has become a JSON string, Spark no longer sees its individual fields as structured columns. Any subsequent operation that needs to inspect those fields may require parsing the JSON again.

A better approach is usually:

read
  ↓
filter
  ↓
join
  ↓
aggregation
  ↓
to_json
  ↓
write

In other words: keep data structured while Spark is transforming it and serialize it only at the boundary where JSON is actually required.

This is both a conceptual and a performance consideration. JSON is a useful interchange format, but it is usually not the representation we want to use for Spark's internal transformations.

What About Jackson or Circe?

General-purpose JSON libraries such as Jackson or Circe still have an important place in a Spark application.

For example, application-level serialization may be preferable when we need:

custom JSON field names or formats,
custom serializers and deserializers,
polymorphic domain models,
domain-specific validation,
precise control over the generated JSON, or
serialization of objects that are not naturally represented as Spark SQL structures.

In such cases, a general-purpose JSON library may be the right abstraction.

The important distinction is therefore not: "Spark JSON is good and Jackson is bad."

Instead, it is:

Spark SQL data
      → use Spark SQL JSON functions

Application/domain objects
      → use an application-level JSON library

Choosing the right abstraction for the layer you are working in can avoid a surprising amount of unnecessary conversion code.

Handling Nulls and Optional Data

Nested data frequently contains optional fields. For example:

case class Customer(
  id: Long,
  name: String,
  address: Option[Address]
)

When serializing nested structures, the handling of null values becomes part of the data contract. Consumers should not assume that every nested object is always present.

The same consideration applies in the opposite direction when using from_json. A missing or invalid value may result in a null structure, so production pipelines should treat the JSON schema and nullability as explicit parts of the contract.

This is another reason why I prefer to keep JSON conversion close to the system boundary: the boundary is exactly where the representation and nullability contract should be defined.

Performance and Schema Stability

JSON is convenient, but it is not free.

Serializing and parsing JSON requires CPU work, and JSON strings are generally less compact than the corresponding structured representation. If a nested structure is repeatedly converted between Spark structures and JSON, the cost can become significant.

This is why I generally recommend avoiding patterns such as:

struct
  ↓
JSON
  ↓
struct
  ↓
JSON
  ↓
struct

when the transformations could have been performed directly on the structured columns.

Schema stability is another important consideration. Once JSON becomes part of an interface between systems, changes to field names, types, nullability, or nesting can affect downstream consumers.

Treat the JSON structure as an API contract rather than as an implementation detail.

Key Learnings

Keep nested data in Spark's native structured representation while transforming it.
Use to_json when a Spark SQL structure needs to cross a JSON boundary.
Use from_json when JSON needs to become a Spark SQL structure again.
Avoid converting Spark data back into Scala objects just to serialize it with a general-purpose JSON library.
Serialize as late as possible, ideally immediately before writing or sending the data to another system.
Use Jackson, Circe or another application-level serializer when you need domain-specific serialization behaviour.
Treat externally visible JSON schemas as contracts and manage their evolution deliberately.

Final Thoughts

JSON is everywhere, which makes it tempting to treat JSON serialization as a generic application problem. In Spark, however, there is an important distinction between serializing a Scala object and serializing data that is already represented inside Spark SQL.

Once the data is inside Spark, I prefer to keep it structured for as long as possible. Spark can filter, join, aggregate and transform nested structures without turning them into strings.

When the data finally needs to leave Spark — for example for a database, Kafka topic or another external system — that is the right time to convert it to JSON.

Keep data structured inside Spark. Serialize it to JSON only at the boundary where another system actually needs JSON.

This simple rule has helped me keep Spark pipelines easier to understand, easier to optimize, and easier to maintain.

← Back to all articles