An Async Email Sending System: A Stack-Independent Design

This post describes a general purpose email sending system, without tying it to any technology. What the email is (an OTP, a notice, a report alert…) is not this system’s problem — that belongs to whoever triggers it. The job here is one clear thing:

Accept the request “send an email to these people, with this template and these parameters”, and carry it out reliably.

Java, Node, Python, Go, .NET — the same principles hold whatever the stack is. There is a mapping table at the end, so you can translate it into your own.

1. Three facts that shape the problem

2. The overall flow

graph TD
    W["Any workflow<br/>login · order · report · alarm"]
    W -->|"in the SAME transaction<br/>as its own domain work:<br/>INSERT status = PENDING"| T[("email_messages<br/>both the work queue<br/>and the send record")]
    T -->|"Model A: DB poll<br/>Model B: relay → broker"| K["Sender worker"]
    K --> C["1 · claim: PENDING → SENDING"]
    C --> R["2 · render the template now"]
    R --> S["3 · send to the provider"]
    S --> D["4 · SENDING → SENT"]
    S -.->|error| RT["retry policy"]
    T -.->|periodic| CL["Cleanup scheduler"]

The key point: the producer never waits. The code that wants to send an email runs one INSERT and it is done. Whatever state the provider is in never touches the producing flow.

3. The data model

EmailMessage
  id
  to              -- recipient(s); SNAPSHOT taken at enqueue time, so the record can
                  --  still say "where did we send it" if the user changes the address
  cc              -- optional list
  bcc             -- optional list
  template        -- template id (e.g. ORDER_CONFIRMED, PASSWORD_RESET)
  parameters      -- the values the template will be filled with (name, code, order no)
                  --  NOT rendered
  attachments     -- list: [{filename, content_type, storage_ref}]
                  --  a reference to the file, NOT the file itself
  status          -- PENDING | SENDING | SENT | FAILED
  attempt_count   -- retry counter
  error_message   -- a short note about the last error (for diagnosis)
  locked_at       -- when a worker claimed the job (for stuck-job detection)
  created_at
  sent_at

Suggested indexes:

Four states are enough: PENDING (waiting), SENDING (a worker took it), SENT (the provider accepted it), FAILED (attempts ran out). Late outcome information like “delivered / bounced” is deliberately out of scope.

4. Atomic enqueue: the record must be born with the trigger

The domain operation that triggers the email (saving the order, creating the OTP) and the INSERT into email_messages must happen in the same transaction.

As long as the email table lives in the same database as the domain data, this guarantee is free: one transaction, two writes. The name for it in the literature is the outbox pattern — here the table already is a natural outbox.

If the trigger is not a database operation at all, a single-row INSERT is atomic by itself; the API returns “accepted” the moment the row commits.

5. Taking the job: an idempotent claim

Whatever model you pick up work with (polling or a broker), the practical guarantee is at-least-once: a job can land in front of more than one worker because of a crash, a redelivery and so on. If two workers send the same email, you lose the duplicate battle right at the start.

So an atomic “claim” before sending is a must:

UPDATE email_messages
SET status = 'SENDING', locked_at = now()
WHERE id = :id AND status = 'PENDING'

-- 1 row affected: this worker won, carry on
-- 0 rows affected: somebody else took it, back off quietly

UPDATE email_messages SET status = 'SENDING', locked_at = now()
WHERE id IN (
  SELECT id FROM email_messages
  WHERE status = 'PENDING'
  ORDER BY created_at
  LIMIT :N
  FOR UPDATE SKIP LOCKED    -- competing workers do not wait for each other
)
RETURNING *

Because the update and its condition are one atomic step (compare-and-swap), only one of two competing workers wins.

6. Jobs stuck in SENDING: stuck-job recovery

The claim above leaves a gap. If a worker takes the job and then dies before finishing it (OOM, a deploy, a lost node), the record stays in SENDING, and no claim looking for status = 'PENDING' will ever pick it up again → an email stuck forever, never sent.

The answer: “a job that was claimed but did not finish in a reasonable time counts as unclaimed” — the database version of the visibility timeout in messaging systems. A scheduled reaper releases them:

UPDATE email_messages
SET status = 'PENDING', locked_at = NULL
WHERE status = 'SENDING'
  AND locked_at < now() - INTERVAL 'X'

Sending an email is a short job (one API call), so X can be aggressive — one or two minutes.

7. The danger window: “I sent it but died before writing SENT”

sequenceDiagram
    participant W as Worker
    participant DB as email_messages
    participant P as Provider
    W->>DB: claim → SENDING
    W->>P: send
    P-->>W: accepted
    Note over W,DB: if the worker dies right here:<br/>the email WENT OUT but the record stays SENDING
    W->>DB: status = SENT

Looking at a record stuck in SENDING, the reaper cannot tell: did the crash happen before the call (the email did not go out → try again), or after it (it went out → trying again means a duplicate)?

You cannot do a side effect on the outside world and your own record in one atomic step. Exactly-once does not exist. The choice is between at-least-once (a rare duplicate) and at-most-once (a rare loss).

You can shrink the risk:

• Some provider APIs accept an idempotency token — they swallow a second request with the same token themselves. Use the record id as the token.
• Narrow the window mechanically: write SENT the moment the call returns, with nothing else in between.

You manage the risk. You cannot remove it.

8. Retry: backoff and a max attempt count

If the provider returns an error, the job does not go straight to FAILED — but it is not retried forever either.

• A counter and a ceiling. Every failed attempt raises attempt_count; when it reaches max_attempts (typically 3–5) the job becomes FAILED and the last error goes into error_message. FAILED records are not deleted — they stay diagnosable, and they get monitored.
• Backoff and jitter. The wait between attempts should grow (say 30 s → 2 min → 10 min), with some randomness on top. Otherwise, during a short provider outage every retry piles in at the same moment and makes the outage last longer.
• A rough error split is enough.

Fine-grained classification (bounce types and so on) is not needed yet. Avoiding the lazy “retry every error 5 times” is enough.

An implementation note: for retry timing you either use locked_at / a separate next_attempt_at column, or — much easier — lean on the retry mechanism of whatever job library you use. Most of them give you this out of the box.

9. Rendering the template at send time

The row holds template + parameters, not rendered HTML. The template is filled in at send time, for two reasons:

• A bug in the template does not leave the queued jobs “frozen with broken content”. You fix the template, and the waiting jobs go out correctly.
• Rows stay small. HTML takes kilobytes, parameters take a few hundred bytes — and table health and poll performance depend directly on that.

A render error (a missing parameter, a template that does not exist) belongs to the permanent error class: a retry will not fix it, so it goes straight to FAILED.

10. Polling or a broker?

There are two models:

For email in particular, the scales usually tip towards DB polling:

• You need the table anyway. The send record, the retry counter, the diagnostic info — all of it needs to be persisted. A broker does not replace the table, it is added on top of it. So the real question is not “table or broker”, it is “is a broker on top of the table worth it?”
• Atomic enqueue is free. In the same-database model, the guarantee from §4 is one transaction. With a broker there is a relay in between (table → relay → broker). It works, but it is a pure extra part.
• The latency is good enough. Poll latency averages half the poll interval, and an interval of a few seconds is usually more than fine for email. And even if you need it faster, a broker is not the only way: mechanisms like Postgres LISTEN/NOTIFY turn polling into push in practice.

11. Retention: a scheduler that cleans old records

email_messages is both a queue and a ledger, so it grows all the time, and unlimited growth slowly weighs down the table, the indexes and the poll query. The answer is a simple scheduled cleanup job:

-- e.g. once a day, at a quiet hour:
DELETE FROM email_messages
WHERE created_at < now() - INTERVAL 'N days'
  AND status IN ('SENT', 'FAILED')
LIMIT :batch_size  -- in chunks instead of one big DELETE (no lock/WAL pressure)

The decisions to make:

• How many days is N? As far back as you want to answer “what did we send and when”. Based on support and audit needs, typically 30–90 days.
• Only the end states are deleted — SENT and FAILED. PENDING and SENDING are never deleted; they are jobs that are still going to run, or running. And if there is a very old PENDING record, that is not a retention matter, it is a signal that something is broken.
• Attachment files have to go too. When a record is deleted, the files it references — the ones the system owns — must be deleted as well, or orphan files pile up in storage.
• Delete or archive. If you do not want to lose the records completely, move them to a cheap archive table or store instead of deleting. Same logic, different target.

Together with the reaper, this scheduler makes up the two “maintenance workers” of the system: one rescues what is stuck, the other clears what is old.

12. Deliberately out of scope for now

The following are not part of this design. It is enough to know they can be added when the need shows up — the current schema does not block any of them.

13. Checklist

• The producer does not wait: an email request is one INSERT, sending is async
• The email record is born in the same transaction as the triggering domain work
• Recipients (to/cc/bcc) are snapshotted at enqueue time
• Attachments are held as references in the row, not as blobs
• An atomic claim before sending — PENDING → SENDING, a conditional UPDATE
• A reaper for jobs stuck in SENDING, plus a client timeout on the provider call
• A deliberate decision about the “sent it but could not record it” window: accept at-least-once, and use the provider’s idempotency token if it has one
• Retry: max attempts plus backoff/jitter; straight to FAILED on 4xx
• The template is rendered at send time; a render error is a permanent error
• The poll-vs-broker decision was made based on the profile — one application → DB poll
• There is a retention scheduler: old SENT/FAILED records and their attachment files are cleaned in chunks
• Basic metrics: the age of the oldest PENDING, the FAILED rate, the send duration

14. Mapping table