How Pepper works

Pepper is one agent that you reach on many channels, with one identity and one memory, built as a strong harness around fast, inexpensive models.

• Channels. Web chat, SMS, email, WhatsApp, phone calls, a command-line tool and native apps.
• Gateway. It verifies each inbound message, finds the person it belongs to, and delivers the reply.
• Agent harness. The orchestrator runs each turn. It loads skills when it needs them, and a skill can start a worker with its own tools. Tools come from 22 feature areas, such as calendar, email and todos, and from servers that you connect.
• Models. Three providers sit behind one interface. Most work runs on the smallest, fastest tier.
• Memory. Rolling summaries of your conversation, a dossier about you, and a search index.
• Background scheduler. It runs the same agent on a timer, for briefs, reminders and watches.
• Rooms. Shared chats with family, friends or a team, where Pepper takes part as a member. A separate service holds them.
• Quality loop. Every turn is recorded. Tests, a replay tool and a nightly model judge read those records.

The design idea: inexpensive models in a strong harness

Every text turn starts on the smallest, fastest model tier that we configure. Voice uses a realtime voice model instead. Small models answer quickly and cost little. Their weak points are predictable: they can stop early, lose track of a long procedure, or answer as if a step happened when it did not. We do not buy our way out of these faults with a larger model. We put the protection in code around the model.

A replay study supports this choice. We replayed recorded turns that had failed, on a small, a medium and a large model. Most failures appeared at every size. The fix that worked at every size was in the harness: tool results that state clearly when there is no data. One failure did depend on model size, so skills that need deep synthesis can ask for a larger tier.

The harness mechanisms that carry the weight:

Bounded loops

A text turn has a round cap, a budget the model can see, repeat-call suppression and a summary at the cap. The life of a turn

Code-enforced gates

Every tool call passes a gate that fails closed. With no person present, code turns sends from your Gmail into drafts. Acting safely

Staged writes

A write that depends on the reply commits only after the reply is delivered. The life of a turn

Small prompts

Skills and workers load tools only when needed, so each prompt holds only what the task uses. Skills and tools

Repair and recovery

A recovery pass rescues empty replies. A repair pass removes garbled text, with strict limits on what it can change. The life of a turn

Routing and tiers

A small tier for most work; larger tiers on request; live comparison of models on real traffic. Models and routing

A cache-friendly prompt

The stable part of the prompt comes first, and the clock goes on the message, so the provider can reuse its cache.

Evidence, not trust

Every turn is recorded. Detectors run in shadow first, and a nightly judge grades real turns. How we check quality

Our design goal is the speed and cost of a small model with no loss of quality. We do not assume we meet it. We check it with the nightly judge, replays on other models, and a fixed rule that decides which quality fixes stay.

Every text message, on any channel, goes through the same loop: build the context, generate, call tools, repeat, then reply.

The gateway records the message before any expensive work, so a retried delivery does not start a second turn. Then the harness builds the prompt from memory, your profile and the list of skills.

Each round is one model generation. If the model asks for tools, the harness runs them in order and gives the results back. Then the next round starts. When the model answers with text and no tool calls, the turn has its reply. The number of rounds in a turn is limited.

A voice conversation, such as a phone call, does not use this loop. It runs in a realtime voice session, and its tool calls pass the same gate.

Writes wait for delivery

A tool runs before the reply exists. Some tools change data because of a reply that the model is about to write. For example, a feed tool marks 22 items as read, and the reply is the briefing of those items. If the reply then fails, you never see the briefing, and the items do not come back.

So these tools stage their writes. When the harness knows that the reply was delivered, it commits all staged writes. If not, it drops all of them.

Guards in the loop

• The gate fails closed. If a policy check fails with an error, the tool call is denied, and the record shows it.
• Wasted rounds don't count toward the limit. A round in which every call was an unknown tool or a refusal is not counted. This applies a small number of times per turn.
• No repeat work. An identical call that already succeeded in this turn does not run again.
• The model sees its budget. Near the cap, the harness tells the model how many rounds remain.
• A capped turn still answers. At the cap, one last generation runs without tools and summarizes the work. It knows which writes completed, so it does not claim one that did not.
• Runtime text is labelled. Each message that the harness adds for the model starts with a fixed line: the user did not write this. Before this rule, a scheduled brief opened with You're right.

Recovery and repair

Sometimes a model returns an empty reply. In one case we observed, a model always returned nothing when a batch of tool results came last in the history. A plain retry failed 8 times in 8. The recovery pass now rewrites the turn as plain text, with no tool structure, and asks again. On the same input, it returned text 24 times in 24.

On email and SMS, a small model reads the final reply before it goes out. It only removes generation defects, such as a duplicated sentence. A deterministic check rejects any edit that changes the length too much, adds more than a few words, or removes a hedge word. If the pass fails, the original reply goes out.

Most work runs on a small, fast model; a larger model runs only where a skill asks for one.

Every model call goes through one internal interface. Behind it are adapters for models from Google, Anthropic and OpenAI. A router picks the adapter from the model name. No other part of the code can import a vendor's SDK. A check in CI fails the build if one does.

A tier names a model and a reasoning effort together. A tier can also hold a weighted mix, so we can compare models on live traffic. The choice is a deterministic hash of the conversation and the date, so one conversation stays on one model for a day. Each turn records which model ran, so the quality reports can compare them. The hash has a useful property: when we add a model or change one weight, only the conversations that must move do move. The mix in each tier changes as we learn.

Failover to another provider is strict: transient errors only, one hop only, and only before the reply starts to stream. Background jobs, such as nightly summaries, use a lower-priority service tier and fall back to the standard tier when that tier is busy. Embeddings and voice do not use the router. Each runs on one provider's model today, and we expect that to change as the industry moves.

A skill is a written procedure; a skill that needs special tools runs as a separate worker, so each prompt holds only the tools that its task needs.

A skill file has a short header and plain-language steps. 23 skills ship with Pepper, and you can write your own. The prompt lists only the name and a one-line description of each skill. When a task matches, the model activates the skill and gets its full steps. The model can choose only from skills that exist.

An illustrative skill file

---
name: github-status
description: Answer a status question about one pull request or issue.
toolsets: [github]
model: medium
---
1. Find the pull request or issue that the user means.
2. Read its current state.
3. Answer in one or two sentences.

Two kinds of skill

A skill with no tool sets of its own is inline. Its steps go to the orchestrator, which follows them. An inline skill can move the rest of the turn to a larger model tier, and it can raise the round cap. It can never lower the cap.

A skill that names tool sets runs as a worker: a separate run of the same loop. The worker gets the skill's steps and only the tools from those sets. A worker cannot start another worker. It has its own round budget.

Why workers matter for small models

Every tool on the orchestrator is in the prompt of every turn. A small model does better with a short list. The GitHub connection alone has about 42 tools. Behind a worker, they enter a prompt only when a GitHub skill runs.

• Direct report. When one worker does the whole job, its report is the reply. This saves one generation.
• Hand back. If a worker needs a tool that it does not have, it hands the step back. The orchestrator gets the data and starts the worker again. This can happen once per turn. The code removes the hand-back tool from the second run: a tool the model cannot see is a stronger rule than an instruction.
• Ask the user. A question ends the run. Your answer starts a normal new turn. An earlier design paused workers and resumed them later. In 30 days it paused 17 times and resumed successfully 4 times, so we removed it.

A skill can also declare the steps that every run must take. After the turn, a detector compares the record with that list and flags a skipped step. This came from a real case: a skill skipped its GitHub check, then told the user that nothing else was open.

Where tools come from

Tools come from 22 feature areas, such as calendar, email, todos, feeds and the dossier. About 80 tools are on the orchestrator. All others sit behind tool sets. The model does not see a tool for an account that you have not connected. Feature areas cannot depend on each other, and a test enforces that rule.

Tools you connect

• GitHub. Pepper uses GitHub's own MCP server. Five GitHub skills, such as the inbox and status questions, run as workers.
• Your MCP servers. You can connect your own MCP servers. When you approve a tool, Pepper stores a fingerprint of its full definition: name, inputs, description and whether it writes. If the server later changes the tool, Pepper holds it back until you approve it again. Otherwise a server could keep an approved name and turn a read into a write.
• The command line. The pepper CLI talks to the same API as the apps. With one flag, it lends the agent read-only access to a folder on your machine. The file tools run on your machine, and the server rejects any tool that does not say it is read-only.

User-written skills can use fewer tool sets than built-in skills. You can share a skill to a room. The room gets a reference, not a copy, and the limits are checked again each time it runs.

Pepper keeps one conversation per person and gives the model a compact view of it on every turn; detailed recall is a search that the model chooses to run.

There is no new chat. Past messages are not replayed to the model. Instead, the prompt holds a memory block: monthly, weekly and daily summaries, and the recent messages that no summary covers yet. Today's messages are never cut. The block covers all channels, and it splits the history into exchanges by gaps in time, not by channel. A texted follow-up to an emailed question is one exchange.

A nightly job writes the summaries. It builds each one from the full record of the turns, never from other summaries, because errors compound. The jobs spread across the hour so that they do not all call the model at once.

The dossier

The dossier is a short document about you, in two halves. You write one half. The model writes the other half, and both halves are in every prompt. During a turn, the model can add a note to its half. Notes only accumulate, so a weekly job rewrites the model's half from the dossier and the recent summaries. Each change keeps a revision. The rewrite is rejected if it is empty, much shorter, or too long.

Search on request

Messages and the documents that Pepper makes are indexed for meaning search. Scheduled runs are left out; the summaries record them instead. The model searches this index when it decides to, and a quoted phrase also runs a keyword search over messages.

In a live conversation, the model confirms before it acts; when no person is present to confirm, code enforces the rules instead.

The persona tells the model to confirm an action that deletes something or that other people can see. That is an instruction, not a guarantee. So every turn is marked as a live conversation, a room or a scheduled run. In a scheduled run, the code changes what the tools are allowed to do.

A background send becomes a draft

• The email tool must always say who sends: Pepper, from its own address, or you, from your Gmail. When Pepper sends from its own address, you are always on CC.
• Mail from your Gmail is a draft by default. In a scheduled run, a request to send it becomes a draft. The tool tells the model, so the reply does not claim a send. The draft appears in chat as a card, where you review and send it.
• Each recipient of a real send must be an address that Pepper can verify. If one is not, the tool refuses rather than guess an address.
• In a scheduled run, Pepper can add a calendar event, but it sends no invitations. It refuses to change or cancel an event that has other attendees.

Other rules in code

• The tool-call gate. Policy checks can allow, deny or change a call. An error in a check counts as a deny. Extra checks can only narrow what is allowed.
• Phone callers. On a direct call to your own Pepper, Pepper answers only a number already verified to an account. Caller ID can be faked, so even then the call starts restricted: no memory in the prompt, and no tool that writes or uses your accounts, until the caller proves who they are. The answering service is different: it answers calls that you forward to it, from anyone. It takes a message in a restricted session, with no access to your memory or accounts. If you turn on booking, it can also see when your calendar is free and book a time.
• Approval follows content. Approval of an MCP tool or an app permission is tied to its exact definition. A change needs a new approval. An app cannot weaken its own confirmation rules.
• Speed bumps. In one turn, the model can cancel two scheduled tasks. A third needs an explicit bulk flag, because a vague turn this off once cancelled all ten of a user's tasks.
• Narrow scopes. Pepper asks Google for read access, plus only the write access it uses: compose for Gmail and events for Calendar.

Scheduled tasks run the same agent on a timer, and a run can stay silent only when the evidence says there is nothing to report.

A task is a prompt with a schedule. Most schedules are coarse, such as mornings, on weekdays, or follow sunrise or sunset.

The scheduler claims each due task with a lease. If a run crashes, the lease expires and the task runs again. A post to a room carries a key that stops a double post. Each task has a small, fixed offset, so tasks set for the same time do not all start at once. Results go to your usual channel, or to a list you choose: chat, email, SMS, WhatsApp or a phone call.

How a run stays silent

A task such as tell me if anything new arrives must be able to say nothing. A run can stay silent only when every lookup it made succeeded and found nothing. Actions that it completed, such as a reaction to a message, do not stop it from staying silent. A run that looked at nothing must report. Every check fails toward delivery: an unwanted message is visible, a swallowed one is not.

Every channel leads to one identity and one memory; rooms add shared data and stricter rules for when Pepper speaks.

Your account is your identity. A phone number or an email address only points to it. So a preference that you give by SMS also applies in email. Each provider's messages are checked for a valid signature before Pepper reads them. A STOP reply blocks SMS, WhatsApp and calls.

Phone calls use a realtime voice model that handles turn-taking and interruptions. Tool calls in a call pass the same gate as text. The call is meant to open with a transcription notice in Pepper's own voice; the model speaks it, so it is an instruction rather than a guarantee. Until the opening ends, Pepper discards the caller's audio. It does not keep it for later.

Rooms

Rooms live in a separate service, and Pepper is a member like a person. Two gates decide when it speaks. First, a fixed rule: Pepper runs only when someone mentions it, replies to it, or follows up soon after it answered them. Saying its name in passing is not enough. Pepper ignores most messages from other agents. Second, the model can still decide to say nothing.

A turn in a room reads the room's data, not only the speaker's. For calendars, each member picks a level for each calendar: nothing, free/busy, titles and times, or everything. Only the member can change it. Pepper's tools cannot. A member who shares a calendar that Pepper cannot read shows as unknown, never as free.

Pepper can follow an ordinary web page, such as a school calendar, and keep your agenda in step with it, at the cost of one model call per change.

You can follow a team, a TV show, a school, a website, a person or a topic. By default, followed events show in your agenda but are not written to your Google calendar, and they do not send invitations. Teams, shows and calendar feeds have structured sources. A plain web page does not, so Pepper reads it with a model.

1. Pepper asks the site whether the page changed. If not, the work stops.
2. It reduces the page to text and compares a fingerprint with the last one. If they match, there is no model call.
3. One model call extracts the events. It must not guess dates, and it must quote the text that supports each event.

A page can move an event to a new date. Pepper matches old and new events by title and nearest date, so a move is an update, not a duplicate. Two weekly events with the same title stay separate. An event that disappears is hidden only after two changed versions of the page leave it out, and it is never deleted. Pepper cancels an event only when the page says it is cancelled.

The calendar also runs detectors that suggest actions: clashes, travel time between events, events found in email, deadlines, and todos that need a time slot.

We record every turn, replay it on demand, grade real turns every night, and keep a quality fix only when the data says it works.

• Records. Each turn is logged for debugging and quality review.
• Replay. A replay tool runs a recorded turn again on another model or setting. Tools return their recorded results, and writes never run again.
• A nightly judge. A model grades recorded turns on seven points, from tool choice to whether the answer is grounded. It grades blind first, then compares with known issues. It opens issues and proposes fixes with failing tests. It never merges.
• Shadow first. A fix that would change a reply first runs in shadow: it records what it would do and changes nothing. After at least 20 firings, it stays at 70% precision or more, and it is deleted below 40%. A four-week limit stops an endless trial.
• Fail open, fail closed. A broken quality fix lets the original reply through. A broken policy check denies the action.

The code base enforces its own architecture in tests. Layers import in one direction only. The test suite has more than 1,000 files and cannot reach the internet: a test that tries fails at once and names the host. A browser test suite drives the real app every night.