Automating API Information Storage with AWS - Technical Deep Dive into Automated API Information Storage System

Managing API information is a big challenge, demanding streamlined solutions for efficiency and reliability. My introduction to automating API information storage showed the need for a sophisticated, automated approach to manage the deluge of API data.

Leveraging AWS's robustness, Docker's agility, and Terraform's precision, I proposed a system to simplify API documentation and discovery.

Here is the repo.

The Role of api_info.json

The automated system lies the api_info.json file, a standardized document that is pivotal in harmonizing API information across repositories. This JSON structure is designed to capture standard details of each API, including its description, version, lifecycle state, and security measures, ensuring a comprehensive database. For instance, the API_version field tracks versioning, enabling developers to access historical and current data seamlessly while API_Security outlines authentication and encryption methods, bolstering security protocols.

{
  "Custody Activity Monitoring": {
    "API_description": "Enhance operational efficiencies with frequent access to data. Monitor financial transactions and account activity with real-time updates.",
    "API_version": "vX",
    "API_Provider_APM_number": "APMXXXXXX",
    "API_Consumers_APM_numbers": [
      "APMXXXXX1",
      "APMXXXXX2",
      "APMXXXXX3"
    ],
    "Lifecycle_State": "Production",
    "Environment": "Production",
    "Visibility": "Public",
    "Intended_Consumers": "External",
    "API_Design": "REST",
    "Request_data_format": "JSON",
    "Response_data_format": [
      "JSON",
      "XML"
    ],
    "API_Owner_LAN_ID": "OwnerIDXXX",
    "Backend_URL": "https://genericdomain.com/api/version/activity",
    "Frontend_URL": "https://apigateway.domain.com/api/version/activity",
    "API_Operations": [
      "GET retrieve activity",
      "POST add activity",
      "PUT update activity"
    ],
    "Dependency_APIs_or_Services": [
      "APMXXXX100",
      "APMXXXX200",
      "APMXXXX300"
    ],
    "API_Gateway_Location": "Location",
    "API_Gateway_instance_info": "API Gateway Instance Details",
    "API_Security": {
      "Authentication": "Authentication Method",
      "Authorization": "Authorization Method",
      "Encryption": [
        "Encryption Method 1",
        "Encryption Method 2"
      ]
    }
  }
}

Setting up the Infrastructure with Terraform

Transitioning to infrastructure, Terraform is the cornerstone of deploying our AWS resources, including the DynamoDB table. The benefits of using Terraform extend beyond automation; it provides a repeatable, error-free deployment process, ensuring consistency across environments. Since this was a PoC, there is very little Terraform:

module "dynamo" {
  source = "terraform-aws-modules/dynamodb-table/aws"

  name = var.environment

  server_side_encryption_enabled = false
  deletion_protection_enabled    = false

  hash_key    = "API_NAME"
  table_class = "STANDARD"

  ttl_enabled = false

  attributes = [{
    name = "API_NAME"
    type = "S"
  }]

  tags = var.tags
}

Dockerizing the Python Application

Dockerization encapsulates the environment required to run our Python automation scripts, guaranteeing consistency and reliability. This segment explores the Dockerfile, detailing the process of building a Docker image that houses our Python application. The advantages of Docker are manifold, offering an isolated environment that mirrors production settings, thereby enhancing testing accuracy and deployment efficiency. In production, this would likely be run off a webhook from the git repository or in the CI/CD pipeline.

FROM python:3.12

WORKDIR /usr/src/app

COPY ./app /usr/src/app/

COPY api_info.json /usr/src/app

RUN pip install --no-cache-dir -r requirements.txt

ARG TABLE_NAME
ENV TABLE_NAME ${TABLE_NAME}

CMD ["python", "./app.py"]

Querying and Using the API Information

A significant advantage of our system is the ease with which developers can query the DynamoDB table for API information. This part showcases practical examples of how developers can leverage this data in application development and API discovery, facilitating a more efficient and informed development process.

For the PoC, I'm using AWS CLI. However, this could sit behind a static frontend, API gateway, and lambda to gain the proper information.

I've shown the integral role each component plays in the overarching system. From the standardization brought about by api_info.json, to the deployment efficiency of Terraform, and Docker and the seamless integration with CI/CD pipelines, each element contributes to a streamlined, efficient management of API documentation. As I refine and enhance this system, I invite feedback and suggestions from the developer community, fostering an environment of continuous improvement and innovation in API management.

Visit my website here.