System Prompt Architecture — Cost Optimization & Implementation Guide

For Developers — Complete Technical Reference

The Problem
Your 3 Ollama Workloads
Research Findings: Can You Use Custom Modelfiles on Ollama Cloud?
7 Architecture Options Compared
Cost Analysis at Different Scales
Recommended Implementation Path
Step-by-Step Developer Instructions

The Problem

The v2 system prompt is 14,253 characters (~3,500 tokens). Currently, this is sent with every API call. With 3 different workloads each using different system prompts, Ollama’s prefix caching provides almost no benefit because every call switches the system prompt (cache miss = full recompute).

The goal: Lowest investment + highest return architecture that:

Avoids re-sending the 14K system prompt on every call
Handles 3 different workloads without cache thrashing
Keeps the system prompt out of the codebase (clean separation)
Allows easy versioning (v2 → v3 → v4)

Your 3 Ollama Workloads

#	Workload	System Prompt Size	Call Frequency	Priority
1	Comment generation	~14,253 chars (~3,500 tokens)	Per post, per user	Highest token cost
2	Post classifier (filters politics, health, personal, job search)	~1,000-2,000 chars (~300-500 tokens)	Per discovered post (highest volume)	Most calls
3	Free tools (10-12 different prompts)	~500-2,000 chars each (~150-500 tokens)	Per tool usage	Variable

The Cache Thrashing Problem

Ollama’s prefix caching is strictly prefix-based. If the system prompt changes between calls, the cache is invalidated:

Call 1: Comment gen     → Cache: [comment-prompt] ✅
Call 2: Classifier      → Cache MISS (different prefix) → Full recompute ❌
Call 3: Comment gen     → Cache MISS again ❌
Call 4: Free tool #3    → Cache MISS again ❌
Call 5: Classifier      → Cache MISS again ❌

Result: Almost zero cache hits. Every call pays full token cost + full compute time.

Research Findings: Can You Use Custom Modelfiles on Ollama Cloud?

Answer: YES — Ollama Cloud (Pro plan and above) supports custom models.

Evidence from research:

Ollama Pricing Page (https://ollama.com/pricing): The Pro plan ($20/mo) explicitly includes “Upload and share private models” as a feature. This means you can create custom models via Modelfiles locally, then push them to the Ollama registry as private models, and use them via the cloud API.
Ollama Cloud Documentation (https://docs.ollama.com/cloud): Cloud models work via the same API. You can access models directly via https://ollama.com/api/chat with your API key. Custom (private) models pushed to your Ollama registry account are accessible the same way.
KodeKloud Tutorial (confirmed workflow): The documented workflow is:
- Create a custom model locally via ollama create growreach-v2 -f Modelfile
- Tag it: ollama copy growreach-v2 your_username/growreach-v2:latest
- Push to registry: ollama push your_username/growreach-v2:latest
- Pull from any server: ollama pull your_username/growreach-v2:latest
- The model page appears at https://ollama.com/your_username/growreach-v2
Ollama Modelfile Documentation (https://docs.ollama.com/modelfile): The SYSTEM instruction in a Modelfile bakes the system prompt into the model. Once created, the model always uses that system prompt — the API caller does not need to send it.
Reddit r/ollama discussions: Multiple users confirmed creating custom Modelfiles with system prompts baked in via ollama create. The custom model behaves like any other model — you just send user messages, no system message needed.
Ollama Cloud API: The direct API at https://ollama.com/api/chat accepts the same format as local Ollama. You specify "model": "your_username/growreach-v2:latest" and send only the user message. The system prompt is already baked in.

Key Finding: How It Works

LOCAL SETUP (one time):
1. Create Modelfile with SYSTEM prompt
2. ollama create growreach-v2 -f Modelfile
3. ollama copy growreach-v2 3rd_rahul/growreach-v2:latest
4. ollama push 3rd_rahul/growreach-v2:latest

PRODUCTION API CALLS:
POST https://ollama.com/api/chat
Authorization: Bearer $OLLAMA_API_KEY
{
  "model": "3rd_rahul/growreach-v2:latest",
  "messages": [
    {"role": "user", "content": "<user prompt only>"}
  ]
}
→ No system prompt needed. It's baked into the model.

Important Caveats:

Your Ollama username: You need to know your ollama.com account username. The model is pushed to username/model-name:tag.
Private models: On the Pro plan ($20/mo), uploaded models are private — only your account can access them. This is exactly what we want.
Cloud model availability: Custom (private) models pushed to the registry should work via the cloud API, but verify this. The Ollama docs say cloud models are “automatically offloaded to Ollama’s cloud service.” Your custom model with a Gemma 4 base should work if Gemma 4 is available as a cloud model. Test this first.
If cloud doesn’t support custom model offload: You can still use Modelfiles by self-hosting Ollama on any server. See Option 6 in the architecture options below.

7 Architecture Options Compared

Option 1: Send Full System Prompt Every Time (Current Default)

How: Every API call includes the full 14K system prompt + user prompt.

{"model": "gemma4:latest", "messages": [{"role": "system", "content": "...14K..."}, {"role": "user", "content": user_prompt}]}

Metric	Value
Tokens per comment	~4,300 input
Cache efficiency	Near 0% (3 workloads thrash cache)
Complexity	Zero
Quality	Highest (full context every time)
Cost at 100 comments/day	~1.3M tokens/day for comment gen alone

Verdict: Works but wasteful. Good for testing, not for production.

Option 2: Prefix Caching Only

How: Rely on Ollama’s automatic prefix caching. Only works if consecutive calls use the same system prompt.

Metric	Value
Tokens per comment	~4,300 (same, caching helps speed not billing)
Cache efficiency	Near 0% with 3 workloads alternating
Complexity	Zero (already enabled)
Quality	Same as Option 1

Verdict: Already happening. Does not solve the problem with 3 workloads.

Option 3: Compress the System Prompt

How: Reduce 14K chars to ~5K while preserving all rules. Less to send each time.

{"model": "gemma4:latest", "messages": [{"role": "system", "content": "...5K compressed..."}, {"role": "user", "content": user_prompt}]}

Metric	Value
Tokens per comment	~2,000 input (55% reduction)
Cache efficiency	Still near 0% (still 3 different prompts)
Complexity	Low (rewrite prompt)
Quality risk	Low-medium (need testing, Gemma 4 may need explicit instructions)
Cost savings	~55% on comment generation tokens

Verdict: Quick win. Do this regardless of which architecture you choose.

Option 4: Two-Stage Pipeline (Classification + Generation)

How: Split into two API calls. First call (cheap model) classifies the post and picks a structure. Second call (generation) uses only the relevant structure’s rules.

Metric	Value
Tokens per comment	~1,300 total (both calls)
Cache efficiency	N/A (different approach)
Complexity	Medium (orchestration code)
Quality risk	Medium (classification accuracy)
Cost savings	~70%

Verdict: Good for scale. Not needed at pre-revenue stage.

Option 5: Fine-Tune a Model

How: Fine-tune Gemma 4 on (user prompt → comment) pairs. System prompt rules learned implicitly.

Metric	Value
Tokens per comment	~800 (user prompt only)
Complexity	High (training pipeline, data collection)
Quality risk	High (inflexible, hard to iterate)
Cost savings	~80%

Verdict: Premature. Wait until v4+ when rules are stable and you have thousands of training examples.

Option 6: Ollama Modelfile (Self-Hosted)

How: Create custom models via Modelfiles. System prompt baked into model config. Developer sends only user prompt.

# One-time setup:
ollama create growreach-comment-v2 -f Modelfile.comment
ollama create growreach-classifier-v1 -f Modelfile.classifier
ollama create growreach-tool-headline -f Modelfile.tool.headline
# etc.

# Production:
{"model": "growreach-comment-v2", "messages": [{"role": "user", "content": user_prompt}]}

Metric	Value
Tokens per comment	~1,000 input (user prompt only)
Cache efficiency	100% (each model keeps its own cache, no thrashing)
Complexity	Low (Modelfile + one command)
Quality	Identical to full prompt
Cost savings	~75% (no system prompt tokens)

Per-workload cache behavior with separate models:

Call 1: growreach-comment-v2     → Cache A: ✅ (warm)
Call 2: growreach-classifier-v1   → Cache B: ✅ (warm)
Call 3: growreach-comment-v2     → Cache A: ✅ HIT!
Call 4: growreach-tool-headline   → Cache C: ✅ (warm)
Call 5: growreach-classifier-v1  → Cache B: ✅ HIT!

Verdict: Best architecture. Solves cache thrashing completely. Clean separation.

Option 7: Modelfile on Ollama Cloud (Push to Registry)

How: Same as Option 6, but push custom models to Ollama’s private registry. Use via cloud API without self-hosting.

# One-time setup (on any machine with Ollama):
ollama create growreach-comment-v2 -f Modelfile.comment
ollama copy growreach-comment-v2 YOUR_OLLAMA_USERNAME/growreach-comment-v2:latest
ollama push YOUR_OLLAMA_USERNAME/growreach-comment-v2:latest
# Repeat for classifier and each tool

# Production (via Ollama Cloud API):
client = Client(
    host="https://ollama.com",
    headers={'Authorization': 'Bearer ' + os.environ['OLLAMA_API_KEY']}
)
response = client.chat(
    model='YOUR_OLLAMA_USERNAME/growreach-comment-v2:latest',
    messages=[{'role': 'user', 'content': user_prompt}]
)

Metric	Value
Tokens per comment	~1,000 input (user prompt only)
Cache efficiency	Depends on Ollama Cloud infra (likely good per-model)
Complexity	Low-Medium (Modelfile + push + API change)
Quality	Identical to full prompt
Cost	$20/mo Pro plan (no self-hosting needed)
Cloud billing	Usage measured by GPU time, not raw tokens. Shorter prompts = less GPU time = less usage consumed.

Critical question to verify: Does Ollama Cloud allow custom (private) model offload to their GPU infrastructure? The Pro plan says “Upload and share private models” but it is unclear if private models get cloud GPU offload or only run locally. Your dev needs to test this.

Test command:

# Create a small test model
echo 'FROM gemma4:latest
SYSTEM "You are a test assistant."' > /tmp/Modelfile.test
ollama create test-custom -f /tmp/Modelfile.test
ollama copy test-custom YOUR_OLLAMA_USERNAME/test-custom:latest
ollama push YOUR_OLLAMA_USERNAME/test-custom:latest
 
# Then try running via cloud API
curl https://ollama.com/api/chat \
  -H "Authorization: Bearer $OLLAMA_API_KEY" \
  -d '{"model": "YOUR_OLLAMA_USERNAME/test-custom:latest", "messages": [{"role": "user", "content": "hello"}]}'

If the test works: Use Modelfiles on Ollama Cloud. Best option. If the test fails (model not found / not available for cloud): Fall back to Option 7-fallback below.

Verdict: Best option IF cloud supports custom model offload. Verify with the test above.

Option 7-Fallback: Compressed Prompt + Ollama Cloud (If Custom Models Don’t Work on Cloud)

If Ollama Cloud does not support running custom (private) models on their GPU infrastructure:

Compress the v2 system prompt from 14K → 5K chars
Send it with every call (Option 3)
Accept the token cost (still 55% less than full 14K)
Stay on $20/mo plan
When scale justifies, self-host Ollama on AWS and use Modelfiles (Option 6)

Cost Analysis

Token Usage Per Approach (Comment Generation Only)

Approach	Tokens/Comment	Tokens/Month (100/day)	Tokens/Month (500/day)	Tokens/Month (1000/day)
Full prompt every time	4,300	12.9M	64.5M	129M
Compressed prompt	2,000	6M	30M	60M
Modelfile (cloud or self-host)	1,000	3M	15M	30M
Two-stage pipeline	1,300	3.9M	19.5M	39M
Fine-tuned model	800	2.4M	12M	24M

Monthly Cost Comparison

Approach	100 comments/day	500/day	1000/day	5000/day
Ollama Cloud $20 (full prompt)	$20 ✅	$20-50 (may need higher plan)	$100+ (need Max)	$500+
Ollama Cloud $20 (compressed)	$20 ✅	$20 ✅	$20-50	$100+
Ollama Cloud $20 (Modelfile)	$20 ✅	$20 ✅	$20 ✅	$20-50
Self-hosted AWS GPU	$60-80	$60-80	$60-80	$60-80
Self-hosted AWS GPU + Modelfile	$60-80	$60-80	$60-80	$60-80 (best at scale)

Break-Even Analysis

Ollama Cloud $20 + Modelfiles	Self-Hosted AWS $60-80
Cheaper up to ~300-500 comments/day	Cheaper at 500+ comments/day
No DevOps	Need GPU instance management
Managed uptime	You own uptime
Easy to iterate (push new model)	Full control

Recommended Path

PHASE 1 (Immediate — Do This Now):
├── Test if Ollama Cloud supports custom model offload (5 min test, see above)
├── If YES → Create 3 Modelfiles, push to registry, update API calls
│   ├── Modelfile.comment (v2 system prompt baked in)
│   ├── Modelfile.classifier (classifier prompt baked in)
│   └── Modelfile.tool-headline, tool-hooks, etc. (per free tool)
│   ├── Cost: $20/mo, ~75% token savings, clean architecture
│   └── Backend change: model name + remove system message
│
├── If NO → Compress system prompt to 5K chars
│   ├── Send compressed system prompt with each call
│   ├── Cost: $20/mo, ~55% token savings
│   └── Backend change: update system prompt string

PHASE 2 (After v2 testing — 2-4 weeks):
├── Compress system prompt to 5K chars regardless (even if Modelfiles work)
├── This gives double savings: Modelfile (no system prompt sent) + compressed (smaller model)
├── Test compressed vs full quality on 50 sample posts
└── If quality holds, update Modelfiles with compressed version

PHASE 3 (At scale — 300+ comments/day):
├── Evaluate self-hosting on AWS GPU spot instance
├── Move comment generation (highest token cost) to self-hosted
├── Keep classifier + free tools on Ollama Cloud (smaller prompts)
├── Or move everything to self-hosted if AWS is cheaper at your volume
└── Consider fine-tuning once rules are stable (v4+)

PHASE 4 (Mature product — 1000+ comments/day):
├── Fine-tune model on real v2/v3 output
├── Eliminate system prompt entirely
├── Two-stage pipeline for further optimization
└── Consider cheaper/smaller models for classifier

Developer Instructions

Step-by-Step: Setting Up Custom Modelfiles on Ollama Cloud

Prerequisites

Ollama Pro plan ($20/mo) — has “Upload and share private models”
Ollama CLI installed on any machine (your VPS, local dev machine, or AWS)
Ollama account username (find at https://ollama.com/settings)
OLLAMA_API_KEY (generate at https://ollama.com/settings/keys)

Step 1: Install Ollama CLI (if not already installed)

curl -fsSL https://ollama.com/install.sh | sh
ollama signin  # Sign in with your ollama.com account

Step 2: Pull the base model

ollama pull gemma4:latest

Step 3: Find your Ollama username

# Your username appears on https://ollama.com/settings
# Or check your public key path to confirm you're signed in
cat ~/.ollama/id_ed25519.pub

Step 4: Create Modelfiles for each workload

Modelfile.comment (for comment generation):

Create a file called Modelfile.comment with this content:

FROM gemma4:latest
 
PARAMETER temperature 0.7
PARAMETER num_ctx 8192
PARAMETER top_p 0.9
 
SYSTEM """
[PASTE THE FULL CONTENT OF v2-system-prompt.md HERE]
"""

Modelfile.classifier (for post classification):

FROM gemma4:latest
 
PARAMETER temperature 0.3
PARAMETER num_ctx 4096
 
SYSTEM """
[PASTE YOUR CLASSIFIER SYSTEM PROMPT HERE]
"""

Modelfile.tool-headline (for LinkedIn Headline Generator):

FROM gemma4:latest
 
PARAMETER temperature 0.8
PARAMETER num_ctx 4096
 
SYSTEM """
[PASTE YOUR HEADLINE GENERATOR SYSTEM PROMPT HERE]
"""

Repeat for each free tool that has a unique system prompt.

Step 5: Create custom models from Modelfiles

ollama create growreach-comment-v2 -f Modelfile.comment
ollama create growreach-classifier-v1 -f Modelfile.classifier
ollama create growreach-tool-headline -f Modelfile.tool.headline
ollama create growreach-tool-hooks -f Modelfile.tool.hooks
# ... one per free tool

Step 6: Test locally first

# Test comment generation
ollama run growreach-comment-v2
# Type a sample user prompt and verify the output follows v2 rules
 
# Test classifier
ollama run growreach-classifier-v1
# Type a sample post and verify it classifies correctly

Step 7: Push models to Ollama registry (private)

# Tag models with your username
ollama copy growreach-comment-v2 YOUR_OLLAMA_USERNAME/growreach-comment-v2:latest
ollama copy growreach-classifier-v1 YOUR_OLLAMA_USERNAME/growreach-classifier-v1:latest
ollama copy growreach-tool-headline YOUR_OLLAMA_USERNAME/growreach-tool-headline:latest
# ... repeat for each tool
 
# Push to registry
ollama push YOUR_OLLAMA_USERNAME/growreach-comment-v2:latest
ollama push YOUR_OLLAMA_USERNAME/growreach-classifier-v1:latest
ollama push YOUR_OLLAMA_USERNAME/growreach-tool-headline:latest
# ... repeat for each tool

Step 8: Test via Ollama Cloud API

# Test comment generation via cloud
curl https://ollama.com/api/chat \
  -H "Authorization: Bearer $OLLAMA_API_KEY" \
  -d '{
    "model": "YOUR_OLLAMA_USERNAME/growreach-comment-v2:latest",
    "messages": [{"role": "user", "content": "Generate a comment for Rahul, founder of 10xers Labs, on a post about AI scaling challenges. Use voice brief: technical founder, direct and analytical."}],
    "stream": false
  }'

If this returns a valid comment: Custom models work on Ollama Cloud. Proceed to Step 9.

If this returns “model not found” or “not available for cloud”: Ollama Cloud does not support custom model offload. Fall back to compressed prompt approach (see Phase 1 “If NO” path above).

Step 9: Update backend API calls

Before (current):

import os
from ollama import Client
 
client = Client(
    host="https://ollama.com",
    headers={'Authorization': 'Bearer ' + os.environ['OLLAMA_API_KEY']}
)
 
# OLD: Sending system + user prompt
response = client.chat(
    model='gemma4:latest',
    messages=[
        {'role': 'system', 'content': SYSTEM_PROMPT_14K},  # Remove this
        {'role': 'user', 'content': user_prompt}
    ]
)

After (with custom models):

import os
from ollama import Client
 
client = Client(
    host="https://ollama.com",
    headers={'Authorization': 'Bearer ' + os.environ['OLLAMA_API_KEY']}
)
 
# NEW: System prompt is baked into the model. Only send user prompt.
response = client.chat(
    model='YOUR_OLLAMA_USERNAME/growreach-comment-v2:latest',  # Changed model name
    messages=[
        {'role': 'user', 'content': user_prompt}  # No system message needed
    ]
)

For the classifier:

response = client.chat(
    model='YOUR_OLLAMA_USERNAME/growreach-classifier-v1:latest',
    messages=[
        {'role': 'user', 'content': post_content}
    ]
)

For free tools:

response = client.chat(
    model='YOUR_OLLAMA_USERNAME/growreach-tool-headline:latest',
    messages=[
        {'role': 'user', 'content': tool_input}
    ]
)

Step 10: Version management

When you update to v3:

# Create new Modelfile with updated system prompt
ollama create growreach-comment-v3 -f Modelfile.comment.v3
 
# Push to registry
ollama copy growreach-comment-v3 YOUR_OLLAMA_USERNAME/growreach-comment-v3:latest
ollama push YOUR_OLLAMA_USERNAME/growreach-comment-v3:latest
 
# Update backend to use v3
# model: 'YOUR_OLLAMA_USERNAME/growreach-comment-v3:latest'
 
# v2 is still available for rollback if needed

Step 11: Save Modelfiles to GitHub

# In the hermes repo
mkdir -p growreach-prompts-v2/modelfiles
cp Modelfile.comment growreach-prompts-v2/modelfiles/
cp Modelfile.classifier growreach-prompts-v2/modelfiles/
cp Modelfile.tool.* growreach-prompts-v2/modelfiles/
 
git add .
git commit -m "Add Ollama Modelfiles for v2 prompts"
git push

Troubleshooting

Problem: “model not found” when calling via cloud API

Ensure you pushed the model: ollama push YOUR_OLLAMA_USERNAME/growreach-comment-v2:latest
Ensure you’re using the correct model name (case-sensitive, includes username)
Ensure your API key has access to the account that owns the model
If cloud doesn’t support custom model offload, use the compressed prompt fallback

Problem: JSON output not being returned properly

Add PARAMETER format json to the Modelfile to force JSON output mode
Or add PARAMETER stop with appropriate stop tokens

Problem: Model quality dropped after compression

Try compressing to 7K instead of 5K
Test which sections can be compressed without quality loss
The 6 structure definitions are the most important section — keep those detailed
The banned phrases list can be condensed (group similar phrases)

Problem: Ollama Cloud usage limits hit too fast

Check your usage at https://ollama.com/settings
Cloud usage is measured by GPU time, not raw tokens
Shorter prompts = less GPU time = less usage consumed
Modelfiles help because the system prompt processing is done at model creation, not per-call
If still hitting limits, consider Max plan ($100/mo) or self-hosting

Summary Decision Matrix

Option	Effort	Token Savings	Cache Solution	Quality	Best For
1. Full prompt every time	Zero	0%	❌ No	Highest	Testing only
2. Prefix caching	Zero	0%	❌ No (3 workloads)	Highest	Already on
3. Compress prompt	Low	55%	❌ No	Low-med risk	Quick win backup
4. Two-stage pipeline	Medium	70%	✅ N/A	Medium risk	Scale stage
5. Fine-tune model	High	80%	✅ N/A	High risk	Mature product
6. Modelfile (self-host)	Low	75%	✅ Yes	Highest	Self-hosting
7. Modelfile (cloud)	Low-med	75%	✅ Yes	Highest	Recommended

The one thing to do today:

Run the 5-minute test in Step 8 above. If it works, your path is clear — create 3 Modelfiles, push to registry, update API calls. If it doesn’t, compress the prompt and stay on cloud until scale justifies self-hosting.

Explorer

architecture-options