Your AI Agent Works in a Demo. Now Make It Survive Production

Building an AI demo has become surprisingly easy.

Connect an LLM API, add a chat interface, give the model a few instructions, and within hours you can have something that looks intelligent.

Then real users arrive.

A customer submits the same request twice. An API times out. A background job runs again after partially succeeding. A user asks the agent to perform an action they are not authorized to perform. Retrieved knowledge is outdated. A webhook arrives three times. The model generates a perfectly formatted but completely incorrect answer.

This is where the difference between an AI demo and an AI system becomes obvious.

At Softquorra, we think about AI agents as software systems first and model-powered components second.

The interesting engineering problem is not:

“How do we call an LLM?”

The real problem is:

“How do we let an AI system interact with business data and business actions without losing reliability, security, observability, or human control?”

This article walks through a practical architecture for building production-grade AI agents.

---

1. Stop Designing Around the Chat Box

Many AI products begin with this architecture:

User
  ↓
Chat UI
  ↓
LLM API
  ↓
Response

This works well for prototypes.

But real business workflows normally look more like this:

User Request
      ↓
Authentication
      ↓
Authorization
      ↓
Intent Detection
      ↓
Context Retrieval
      ↓
Agent Reasoning
      ↓
Tool Selection
      ↓
Validation
      ↓
Business Action
      ↓
Audit Logging
      ↓
Response

And that is still a simplified version.

The model should not be the architecture.

It should be one component inside the architecture.

A production AI application still requires the same engineering principles as any serious SaaS platform:

• authentication

• authorization

• data validation

• transaction handling

• retries

• background processing

• monitoring

• rate limiting

• audit logs

• testing

• error recovery

AI introduces additional uncertainty, but it does not remove traditional software engineering requirements.

---

2. Separate Reasoning From Execution

One of the most important architectural decisions is separating what the model decides from what the system executes.

Imagine an AI sales assistant.

The user says:

Send a follow-up email to every lead
who opened our previous campaign.

A dangerous architecture would allow the model to directly query the database and send thousands of emails.

A safer architecture looks like this:

User
 ↓
Agent
 ↓
Generate Action Proposal
 ↓
Permission Check
 ↓
Validate Parameters
 ↓
Human Approval
 ↓
Execution Service
 ↓
Audit Log

The model can propose:

{
  "action": "send_follow_up_campaign",
  "segment": "opened_previous_campaign",
  "campaignId": 812
}

But the application determines whether that action can actually happen.

This creates a critical architectural boundary:

LLM decides intent
Application decides permission
Application executes action

The LLM should never become your authorization system.

---

3. Treat Tools Like Internal APIs

Modern AI agents often use tools.

Examples include:

searchCustomers()
createLead()
sendEmail()
createInvoice()
updateCRM()
scheduleMeeting()
generateReport()

A tool should behave like a secure API endpoint.

Consider a simplified TypeScript example:

interface CreateLeadInput {
  companyName: string;
  email: string;
  source: string;
}

async function createLead(
  input: CreateLeadInput,
  user: AuthenticatedUser
) {
  if (!user.permissions.includes('lead:create')) {
    throw new ForbiddenException();
  }

  const validatedInput = createLeadSchema.parse(input);

  return leadService.create({
    ...validatedInput,
    createdBy: user.id,
  });
}

Notice what is missing.

The model does not decide whether the user is allowed to create a lead.

The application does.

Every AI tool should ideally include:

Authentication
Authorization
Input validation
Business validation
Execution
Logging
Error handling

This prevents a prompt from bypassing business rules that already exist elsewhere in the application.

---

4. Use Structured Outputs Instead of Parsing Natural Language

Suppose your agent returns:

I think we should create a customer named John
using [email protected].

Now your backend needs to guess what part of that sentence contains the actual action.

That becomes fragile quickly.

Instead, require structured output.

For example:

{
  "action": "CREATE_CUSTOMER",
  "parameters": {
    "name": "John",
    "email": "[email protected]"
  },
  "confidence": 0.91
}

Then validate it using something like Zod:

const agentActionSchema = z.object({
  action: z.enum([
    'CREATE_CUSTOMER',
    'UPDATE_CUSTOMER',
    'SEND_EMAIL'
  ]),
  parameters: z.record(z.any()),
  confidence: z.number().min(0).max(1)
});

Your agent becomes much easier to integrate into deterministic software.

The model handles interpretation.

Your code handles contracts.

---

5. Introduce an Agent Orchestration Layer

Once agents perform more than one action, orchestration becomes necessary.

Instead of:

Controller
   ↓
LLM

consider:

Controller
   ↓
Agent Orchestrator
   ↓
 ┌─────────────────────┐
 │ Context Builder     │
 │ Model Gateway       │
 │ Tool Registry       │
 │ Policy Engine       │
 │ Approval Manager    │
 │ Memory Manager      │
 │ Audit Logger        │
 └─────────────────────┘

The orchestrator owns the workflow.

For example:

async function executeAgentTask(task: AgentTask) {
  const context = await contextBuilder.build(task);

  const decision = await modelGateway.reason({
    task,
    context,
  });

  const action = actionSchema.parse(decision);

  await policyEngine.validate(action, task.user);

  if (requiresApproval(action)) {
    return approvalManager.createPendingAction(action);
  }

  return toolRegistry.execute(action);
}

This makes it possible to replace or upgrade individual components without rebuilding the entire system.

---

6. Do Not Put Long-Running Work Inside HTTP Requests

AI operations can be slow.

External APIs can be even slower.

Imagine an endpoint that:

1. retrieves 5,000 leads

2. analyzes each lead

3. generates personalized messages

4. sends emails

5. updates the CRM

6. calculates analytics

Trying to complete everything inside:

POST /campaign/start

is a reliability problem waiting to happen.

Instead:

POST /campaign/start
        ↓
Create Campaign Job
        ↓
Queue
        ↓
Worker
        ↓
Process Leads
        ↓
Persist Progress
        ↓
Update Campaign Status

The API can respond immediately:

{
  "campaignId": 1421,
  "status": "queued"
}

The worker processes the expensive operation asynchronously.

Useful technologies could include:

AWS SQS
BullMQ
RabbitMQ
Kafka
Google Pub/Sub
Azure Service Bus

The technology is less important than the architecture.

---

7. Design Every Background Job for Retries

Queues create another problem:

the same job may run more than once.

Assume a worker sends a payment receipt.

The first execution succeeds but crashes before acknowledging the queue message.

The queue retries it.

Without protection:

Customer receives receipt #1
Customer receives receipt #2

For email, that is annoying.

For payments, duplicate execution can become much worse.

This is why production workflows need idempotency.

For example:

const existingExecution =
  await executionRepository.findByIdempotencyKey(
    job.idempotencyKey
  );

if (existingExecution) {
  return existingExecution.result;
}

You can also enforce this in PostgreSQL:

CREATE UNIQUE INDEX
idx_agent_execution_idempotency
ON agent_executions(idempotency_key);

The rule becomes:

Same logical request
+
Same idempotency key
=
One business operation

Retries should be safe by design.

---

8. Store Agent Execution State

A serious AI workflow should not disappear after the HTTP response.

Persist its lifecycle.

For example:

agent_execution

id
tenant_id
agent_id
user_id
input
status
model
prompt_version
tool_calls
output
error
started_at
completed_at

Possible statuses:

QUEUED
RUNNING
WAITING_FOR_APPROVAL
COMPLETED
FAILED
CANCELLED

Now your application can answer important questions:

What happened?

Who started it?

Which model was used?

Which tools were called?

Which prompt version generated the decision?

Did someone approve the action?

Why did it fail?

That becomes extremely valuable when debugging production AI behavior.

---

9. Human-in-the-Loop Is an Architecture Pattern

“Human approval” should not be a button added later.

It should be part of your workflow model.

For example:

AI generates customer reply
        ↓
Risk Evaluation
        ↓
Low Risk ───────────→ Send Automatically

High Risk
   ↓
Pending Approval
   ↓
Human Reviews
   ↓
Approve / Reject

Actions that may deserve approval include:

• refunds

• account deletion

• financial transactions

• customer-facing legal communication

• large email campaigns

• changes to permissions

• destructive database operations

• publishing content publicly

Your database could contain:

agent_approvals

id
execution_id
requested_action
status
requested_at
reviewed_at
reviewed_by

This gives your organization control without removing the efficiency provided by AI.

---

10. Retrieval-Augmented Generation Is Mostly a Data Problem

RAG is often introduced as:

Documents
   ↓
Embeddings
   ↓
Vector DB
   ↓
LLM

But production retrieval involves more than similarity search.

You need to answer:

Which documents may this user access?

Which tenant owns this document?

Is this document still valid?

When was it updated?

Should archived documents be retrieved?

Which version has priority?

Can confidential content leave the system?

Your retrieval query might therefore include filters such as:

{
  tenantId: user.tenantId,
  departmentId: user.departmentId,
  status: 'ACTIVE',
  accessLevel: {
    $in: user.allowedAccessLevels
  }
}

Retrieval must respect the same permission boundaries as the rest of the product.

Otherwise, you can build a technically impressive RAG system that becomes a security vulnerability.

---

11. Multi-Tenant AI Requires Data Isolation

For SaaS products, this becomes even more important.

Imagine:

Tenant A
- customers
- documents
- conversations

Tenant B
- customers
- documents
- conversations

An agent running for Tenant A must never retrieve Tenant B's content.

Every query should therefore be scoped.

A simple pattern:

await repository.find({
  where: {
    tenantId: authenticatedUser.tenantId
  }
});

For stronger isolation, applications may use:

• PostgreSQL Row-Level Security

• separate schemas

• separate databases

• tenant-aware repositories

• tenant-scoped vector indexes

There is no universal strategy.

But there should always be an explicit strategy.

---

12. Version Your Prompts

Software engineers version code.

AI systems should also version prompts.

Imagine changing:

You are a helpful customer support assistant.

to a much larger prompt containing new rules.

Suddenly answer quality decreases.

Without versioning, debugging becomes difficult because you cannot determine which instructions generated a response.

Store something like:

prompt_version = support-agent-v12

along with every execution.

Then analytics can tell you:

v10 → 82% accepted replies
v11 → 76% accepted replies
v12 → 91% accepted replies

Prompt engineering becomes measurable instead of subjective.

---

13. Build a Model Gateway

Applications should avoid scattering direct model calls throughout the codebase.

Instead of:

openai.chat(...)

inside dozens of services, create a centralized abstraction.

For example:

interface ModelGateway {
  generate(
    request: ModelRequest
  ): Promise<ModelResponse>;
}

Then implementations could include:

OpenAIModelGateway
AnthropicModelGateway
GeminiModelGateway
LocalModelGateway

Your business logic talks to:

ModelGateway

rather than directly to one vendor.

A gateway can centralize:

• model selection

• token limits

• retries

• timeouts

• logging

• cost tracking

• fallbacks

• structured outputs

• safety configuration

It also makes model migration significantly easier.

---

14. Observability Matters More With AI

Traditional monitoring asks:

Did the request fail?
How long did it take?

AI monitoring needs additional questions:

Which model responded?

How many tokens were consumed?

How much did the request cost?

Which documents were retrieved?

Which tools were called?

Was the response approved?

Was it regenerated?

Did the customer accept the answer?

Which prompt version was used?

A useful event might look like:

{
  "executionId": "exec_91827",
  "agent": "support-agent",
  "model": "model-x",
  "latencyMs": 2840,
  "inputTokens": 1840,
  "outputTokens": 412,
  "retrievedDocuments": 4,
  "toolCalls": 2,
  "status": "completed"
}

Once this information is centralized, dashboards become possible.

You can measure:

Success Rate
Failure Rate
Average Latency
Average Cost
Human Approval Rate
Tool Failure Rate
Customer Acceptance Rate

Without observability, improving an agent becomes guesswork.

---

15. Assume External Services Will Fail

Your AI product may depend on:

LLM provider
CRM
Stripe
Email provider
WhatsApp
Google Calendar
Slack
ERP
Vector database
Internal APIs

Every dependency will eventually fail.

A robust integration therefore needs:

Timeout
Retry
Backoff
Idempotency
Logging
Dead-letter handling
Recovery

A retry strategy might be:

Attempt 1 → immediately
Attempt 2 → 5 seconds
Attempt 3 → 30 seconds
Attempt 4 → 2 minutes
Failure → dead-letter queue

Importantly, not every error should be retried.

HTTP 429 → retry
HTTP 503 → retry
Network timeout → retry

HTTP 401 → configuration problem
HTTP 403 → permission problem
HTTP 400 → likely invalid request

Intelligent retry behavior prevents temporary failures from becoming permanent data problems.

---

16. A Practical Production Architecture

Putting these ideas together produces something like:

                         ┌──────────────┐
                         │   Web App    │
                         └──────┬───────┘
                                │
                         ┌──────▼───────┐
                         │ API Gateway  │
                         └──────┬───────┘
                                │
                    ┌───────────▼───────────┐
                    │ Authentication / RBAC │
                    └───────────┬───────────┘
                                │
                     ┌──────────▼─────────┐
                     │ Agent Orchestrator │
                     └──────────┬─────────┘
                                │
          ┌─────────────────────┼─────────────────────┐
          │                     │                     │
    ┌─────▼─────┐        ┌──────▼─────┐       ┌──────▼─────┐
    │ Retrieval │        │Model Gateway│       │Tool Registry│
    └─────┬─────┘        └──────┬─────┘       └──────┬─────┘
          │                     │                     │
    Vector / DB             LLM Provider        Business APIs
          │                                           │
          └─────────────────────┬─────────────────────┘
                                │
                         ┌──────▼──────┐
                         │Policy Engine│
                         └──────┬──────┘
                                │
                      ┌─────────▼─────────┐
                      │ Approval Workflow │
                      └─────────┬─────────┘
                                │
                         ┌──────▼───────┐
                         │ Queue / Jobs │
                         └──────┬───────┘
                                │
                         ┌──────▼───────┐
                         │   Workers    │
                         └──────┬───────┘
                                │
                    ┌───────────▼───────────┐
                    │ DB + Audit + Metrics  │
                    └───────────────────────┘

It looks more complicated than:

Frontend → LLM

because the real-world problem is more complicated.

The additional layers are what make the system controllable.

---

17. What We Optimize for at Softquorra

A useful AI system should not exist just because AI is available.

It should improve a measurable workflow.

Before designing an agent, we prefer to identify:

What work happens repeatedly?

Where does the team lose time?

Which decisions require human judgment?

Which decisions can safely be automated?

What systems already contain the required data?

What happens when automation fails?

How will success be measured?

Then the workflow can be divided into three categories.

Deterministic

Normal software should handle it.

Validation
Authentication
Calculations
Database constraints
Permissions
Payment state

Probabilistic

AI may be useful.

Classification
Summarization
Content generation
Intent detection
Document interpretation
Lead qualification

Sensitive

AI may assist, but humans should remain involved.

Financial actions
Destructive operations
Legal communication
Important customer decisions
Large public actions

This separation is often more valuable than choosing the newest model.

---

18. The Production Checklist

Before calling an AI agent production-ready, ask:

Security

• Are tools permission-aware?

• Is tenant data isolated?

• Can retrieved data leak between users?

• Are secrets stored securely?

Reliability

• Are long jobs asynchronous?

• Are jobs idempotent?

• Are retries safe?

• Are failed jobs recoverable?

AI

• Are prompts versioned?

• Are outputs validated?

• Can the model call only approved tools?

• Are important actions protected by approval?

Observability

• Are model calls logged?

• Are tool calls logged?

• Can we calculate cost?

• Can we measure successful outcomes?

Data

• Is retrieved knowledge current?

• Are access rules enforced?

• Can users trace where an answer came from?

Operations

• Can administrators disable an agent?

• Can a failed workflow be retried?

• Can a human inspect execution history?

• Can risky actions be cancelled?

If several of these answers are “no,” you probably have an AI prototype rather than a production AI system.

---

Final Thoughts

The hardest part of building an AI product is rarely connecting to the model.

The harder problem is surrounding a probabilistic model with deterministic engineering.

Production AI needs:

AI reasoning
+
Software architecture
+
Permission boundaries
+
Reliable integrations
+
Background processing
+
Human oversight
+
Observability

The model may generate the intelligence.

The architecture creates the trust.

That distinction matters whether you are building an AI support agent, marketing automation platform, internal copilot, SaaS product, lead qualification system, document-processing pipeline, or autonomous workflow.

At Softquorra, this is the engineering mindset we apply when designing AI agents, SaaS platforms, integrations, and business automation systems: start with the actual workflow, identify what should and should not be automated, then build the infrastructure that lets AI operate safely inside it.

Because getting an AI agent to answer a prompt is easy.

Getting one to work reliably when customers, permissions, APIs, queues, failures, money, and production data are involved is the real engineering challenge.