Skip to main content
Platform serviceReal-time event & job gateway

AI/ML Gateway

Long-running work — reports, AI calls, bulk jobs — runs in the background and notifies users live, so the core API never blocks.

In plain words

Some tasks, like generating a big report or calling an AI model, take a minute or more. Instead of making people stare at a spinner, the gateway takes the job, works on it in the background, and pings the user's screen the moment it's done.

Interactive preview

How AI/ML Gateway works

Interactive · illustrative sample data

The challenge

Holding an HTTP request open for 30–90 seconds risks load-balancer timeouts, duplicate work when users refresh, and a clogged API.

Our approach

Enqueue and notify: accept the job immediately, process it asynchronously from a Redis queue, then push the result to the right browser over WebSockets.

Step by step

From first click to finished job

  1. 1

    User starts a long task

    The main app enqueues it and returns a job ID immediately.

  2. 2

    Worker processes the job

    From the Redis queue, with correlation IDs for tracing.

  3. 3

    Browser gets notified

    A Socket.IO event delivers progress and completion live.

Capabilities

What it does

Redis job queues

Durable job storage with retries.

Socket.IO rooms

Per-user/session rooms so notifications never leak across accounts.

Idempotent handlers

Retries don't duplicate side effects.

Graceful degradation

If sockets drop, clients can poll job status.

Engineering

Key decisions

Respect request lifetimes

Heavy work never runs inline in the core API.

Architecture

Tech stack

Service

  • Node.js
  • Express
  • TypeScript

Realtime & jobs

  • Socket.IO
  • Redis
  • Sequelize

Outcome

What it delivers

  • Core API stays responsive while heavy jobs run in the background
  • Users see live progress instead of timeouts

Deep dives

Read the engineering notes

Services behind it