From Mainframe to Microservice: Modernizing COBOL Data Processing

Introduction

Legacy systems are the heartbeat of many enterprises, particularly in finance, government, and healthcare. At the center of those systems lies COBOL—a language synonymous with reliability and decades of business logic. But as the demand for agility and integration rises, organizations need to rethink how COBOL workloads operate in a modern context. This post walks through the architecture and implementation of a microservice-based pipeline for COBOL data processing. It replaces monolithic, batch-driven mainframe workflows with modular, containerized services that handle ingestion, execution, enrichment, and storage—plus a path for ML-based error prediction downstream.

The Full Pipeline: Ingesting Files, Running COBOL, Enriching Data

In our modern architecture, the COBOL process is just one service in a larger, automated pipeline. The flow looks like this:

1. File Ingestion Service
   A microservice or scheduled job downloads or receives raw input files (CSV or fixed-width format) and places them into an EFS-mounted directory accessible by the COBOL execution pod. This ingestion can come from APIs, SFTP, or direct user uploads, depending on enterprise requirements.

2. COBOL Execution Service
   A containerized service running GnuCOBOL compiles and runs the legacy COBOL code against the files on EFS. If successful, enriched records are saved to a separate subdirectory for downstream processing. If any errors are encountered (invalid formats, data exceptions), they are logged to a file in a designated logs/ folder.

3. Data Enrichment & Transformation
   A microservice picks up the enriched outputs and creates SQL inserts for PostgreSQL. The goal is to standardize the data into a format modern tools can easily consume, tag it with metadata, and ensure it conforms to the required schema.

The entire pipeline is orchestrated with Kubernetes Jobs or CronJobs for scheduled runs, with services communicating via mounted volumes and async triggers (e.g., watching a directory or publishing to an internal SNS topic).

Shell Scripts, Job Orchestration, and Service Communication

Unlike traditional JCL job chains, our microservice-based approach uses shell scripts and Kubernetes-native features to coordinate execution. Each COBOL job pod starts with a script like this:

#!/bin/bash
set -e

echo "Compiling COBOL..."
cobc -x -free /mnt/data/source/TransformCSV.cbl -o /mnt/data/bin/TransformCSV

echo "Running COBOL..."
/mnt/data/bin/TransformCSV

This script compiles the COBOL code, executes it, and expects input/output to be on a mounted EFS volume (e.g., /mnt/data/input and /mnt/data/output). By containerizing the logic and mounting shared storage, we decouple the execution from any one node or persistent machine.

Job orchestration is handled via Kubernetes Jobs, with restartPolicy: Never, ensuring that each run is independent and idempotent. If a job fails, its pod logs can be reviewed, and its output/error artifacts will still exist on EFS for debugging.

Service communication is minimal and file-based, but can be extended using S3, SQS, or an internal metadata service. The enrichment job polls or watches EFS and picks up completed jobs based on naming conventions or success markers (e.g., writing a .done file after a run).

Interfacing Between COBOL Output and Modern Services

COBOL was designed in a world of mainframes, not REST APIs or cloud services. Its outputs are usually flat files—either CSV, fixed-width text, or simple logs. Modern services expect structured, validated, typed data. That’s where our enrichment/transformation layer comes in.

A Python-based transformer service runs on a regular interval (or is triggered by file drops). It processes COBOL outputs like this:

• Validates required fields and record counts.
• Transforms strings into typed fields (e.g., dates, floats).
• Wraps each record in a JSON schema or prepares a batched SQL statement.
• Pushes enriched records into a PostgreSQL-compatible service or stores them in S3 for future analysis.

If the COBOL output indicates an error (e.g., missing field, invalid input), the transformer writes a corresponding JSON error document to logs/ with metadata such as timestamp, source file, and failure reason. These JSONs become training data for SageMaker down the road.

Transforming COBOL Records for PostgreSQL and Error Handling

COBOL applications typically use flat-file formats with no schema enforcement. To store data in PostgreSQL, we first define a target schema:

CREATE TABLE customer_data (
  id UUID PRIMARY KEY,
  name TEXT,
  age INT,
  balance DECIMAL,
  created_at TIMESTAMP DEFAULT now()
);

Our transformer maps each COBOL field to a column in this table, handling edge cases (empty strings, overflows) and ensuring type safety. Here’s a sample Python snippet that does that mapping:

import csv
import psycopg2

def transform_and_insert(filepath):
    conn = psycopg2.connect(...)
    cur = conn.cursor()

    with open(filepath, newline='') as csvfile:
        reader = csv.DictReader(csvfile)
        for row in reader:
            cur.execute(
                "INSERT INTO customer_data (id, name, age, balance) VALUES (%s, %s, %s, %s)",
                (row['id'], row['name'], int(row['age']), float(row['balance']))
            )

    conn.commit()
    cur.close()
    conn.close()

Errors encountered during parsing or inserts are logged to individual JSON files like:

{
  "error": "ValueError: could not convert string to float",
  "source_file": "customer_batch_09122025.csv",
  "record": {
    "id": "123",
    "name": "John Doe",
    "age": "not_available",
    "balance": "abc123"
  },
  "timestamp": "2025-09-12T14:33:00Z"
}

These files are stored in an S3 bucket and used by an ML model to train on what types of records are prone to failure.

Conclusion

This pipeline is a living example of how legacy systems don’t need to be buried—they can be integrated, extended, and made smarter using cloud-native patterns. COBOL still has a role in the enterprise, especially when paired with containers, Kubernetes, and microservices that wrap it in a modern runtime.

We took a brittle, batch COBOL pipeline and turned it into a scalable, observable, and ML-ready platform. Each service is isolated, testable, and built with the same tools we use for Python, Go, or Node.js apps. And by leveraging AWS services like EFS, S3, PostgreSQL, and SageMaker, we now have full visibility, traceability, and predictive power over every step of the legacy pipeline.