Document processing (Job API)
Job lifecycle
JOB_STATUS_NEW → JOB_STATUS_FILE_PROCESSING → JOB_STATUS_OCR → JOB_STATUS_LLM → JOB_STATUS_COMPLETE | JOB_STATUS_PARTIAL | JOB_STATUS_FAILED
- JOB_STATUS_NEW — the job has been created and queued.
- JOB_STATUS_FILE_PROCESSING — files are being downloaded, archives unpacked, and formats normalized into a processable form. Can transition directly to
JOB_STATUS_FAILEDif all sources are unreachable or a file cap is exceeded (reason in the job’serrorfield). - JOB_STATUS_OCR — text recognition for each file.
- JOB_STATUS_LLM — the recognized text is sent to the model with your prompts.
- JOB_STATUS_COMPLETE — no errors at the OCR or LLM stage.
- JOB_STATUS_PARTIAL — at least one successful model (LLM) answer, but also at least one error at the OCR or LLM stage (check the file-level errors in the result), or an answer that doesn’t conform to
responseSchema(see “Structured output”). - JOB_STATUS_FAILED — errors prevented any file from reaching a successful model answer: either a failure during file download/unpacking (reason in the job-level
errorfield;ocr[]/llm[]are empty), or no file reached a successful result at the OCR or LLM stage.
Processing is asynchronous: poll the job status via GET /v1/jobs/{id} until the job reaches a terminal status.
Uploading files
You can specify a job’s source in two ways:
- Public URL — pass the URL in
sourceUrlswhen creating the job. ChunkChef downloads the file (download timeout: 30 s, up to 3 redirects). - Direct upload — upload the file, then use the returned URL in
sourceUrls:POST /v1/jobs/upload(multipart/form-data) — uploads a single file over HTTP. This is a standalone HTTP endpoint (not grpc-gateway). The response is JSON in snake_case:{"url": "…", "name": "…", "size_bytes": 12345}. This endpoint’s error body is{"code": <int>, "message": "…"}, with nodetailsarray. The size limit is set by config (grpc.maxRecvMsgBytes; 20 MiB in the current deployment), not a code default.- gRPC method
Upload(client-streaming) — a streaming upload (single-message limit: 20 MiB).
Prompts and data extraction
Data extraction is driven by text prompts, not by a schema. In the prompts field you pass an array of instructions. At the LLM stage, each file’s recognized text is taken, split into chunks if needed, and sent to the model together with your prompt. The model’s answer is returned as text per file (and per chunk, if the file was split). Optionally, add responseSchema — a JSON Schema — to have ChunkChef validate the answer’s shape after the call; see “Structured output” below.
Chunk splitting is structure-aware: the text is cut along the boundaries of its format’s structure (Markdown, HTML, XML, or plain) — tables are not torn mid-row (and if a table doesn’t fit whole, its header row is repeated in each chunk), and section-heading context is preserved. The chunk size in tokens is set by neural.chunkBudgetTokens (see “Connecting a model”); if unset, the service’s conservative default applies. Each chunk is a separate model call with a full copy of the prompt and a separate llm[] row (chunkIndex / chunkTotal). Reassembling the answer from chunks in chunkIndex order is done on your side by default. Optionally, ChunkChef can do that reassembly for you: set merge.enabled to have the service combine the per-chunk answers into one whole-document result — see “Merging chunked results” below.
To get structured data, ask for it directly in the prompt — for example: “Return the result as JSON with the following fields: …”. Your prompt and the model you chose determine the validity and shape of the JSON; ChunkChef neither imposes nor validates a schema, unless you opt in with responseSchema (see “Structured output” below).
Prompt tips:
- list the fields you need explicitly and unambiguously;
- specify the output format right in the prompt text;
- keep the prompt size limit in mind — 64 KiB (see “Limits”).
Note. The <…> placeholders in the templates below mark spots for you to fill in: ChunkChef does not substitute them automatically — the prompt is sent to the model exactly as written. Your prompt and the model you choose determine the output structure; there’s no server-side schema validation unless you set responseSchema (see “Structured output”).
Prompt quality drives your results. How well extraction works depends as much on your prompt as on the model you choose — often more. A vague prompt produces vague output even on a top-tier model, while a precise, well-structured prompt gets reliable results even from smaller, cheaper models. Treat the template below as a starting point, not a finished prompt: take it, describe your document type, your task, and the exact output you need, then ask a capable model to turn it into a prompt tailored to your case — keeping this structure while sharpening the rules, edge cases, and output validation for your data. The meta-prompt for that is at the end of this section.
Example: extracting invoice / order / receipt fields
prompts[0] text:
TASK
Extract structured fields from a single procurement/accounting document and return them strictly as JSON.
IMPORTANT
Your answer must contain ONLY JSON. Do not add any comments, explanations, or surrounding text before or after the JSON.
1. INPUT
The recognized text of a single document follows the "---" marker below (the OCR output is HTML). It is the only source. The document may be an <type: invoice / purchase order / receipt> and may contain stamps, signatures, and multi-row line-item tables. The text may be truncated or split into chunks — work only with the text you are given and never assume content you cannot see.
2. OUTPUT JSON FORMAT
Return a single JSON object matching this schema (the inline comments are explanatory — do not include them in the output):
{
"doc_type": "string", // one of: invoice | purchase_order | receipt | unknown
"number": "string",
"date": "string", // ISO 8601: YYYY-MM-DD
"supplier": {
"name": "string",
"tax_id": "string" // e.g. US EIN or EU VAT ID, as printed
},
"items": [
{ "name": "string", "qty": number, "price": number, "amount": number }
],
"total": number,
"currency": "string" // ISO 4217, e.g. USD, EUR
}
3. EXTRACTION RULES
- doc_type: classify from the title, headers, and content. If it is none of the listed types, set "unknown" and still fill any fields you can.
- number / date: the document's own number and issue date. Convert the date to ISO 8601 (YYYY-MM-DD).
- supplier: the selling/issuing party, not the buyer. tax_id: the supplier's tax identifier, exactly as printed.
- items: one object per line item, in document order. Keep "name" exactly as written, including specifications and units that identify the item.
- qty / price / amount / total: return as JSON numbers — strip thousands separators and currency symbols, use a dot as the decimal separator ("1,200.50" -> 1200.5).
- currency: ISO 4217 code. If only a symbol is present, map it ("$" -> "USD", "€" -> "EUR"). If it cannot be determined, use null.
4. PROCESSING REQUIREMENTS
- Use ONLY the provided document text. Do not add external knowledge or infer values that are not present.
- Do NOT guess, complete, or reformat values beyond the normalization explicitly required above.
- Field not found -> null for scalars (including supplier sub-fields), [] for "items". Never drop a schema key.
- Analyze the entire document, including tables and appendices. A reference to an external attachment is not a line item.
- Be literal and deterministic: the same input must always produce the same output.
5. RESPONSE FORMAT
- Return ONLY the valid JSON object described above.
- No markdown, no code fences, no text before or after the JSON.
REMEMBER
Your answer must start with "{" and end with "}". Nothing else. If the document is not one of the expected types, return the schema with "doc_type": "unknown" and whatever fields you could extract.
JSON envelope (primary — on the verified Xiaomi mimo-v2-flash):
{
"sourceUrls": ["<YOUR_FILE_URL>"],
"title": "Invoice <number>",
"prompts": ["<THE ENTIRE TEMPLATE ABOVE, AS A SINGLE STRING>"],
"ocr": { "provider": "NEURAL_CLIENT_TYPE_MISTRAL", "model": "mistral-ocr-latest", "providerKey": "<YOUR_KEY>" },
"neural": {
"type": "NEURAL_CLIENT_TYPE_XIAOMI",
"model": "mimo-v2-flash",
"apiKey": "<YOUR_PROVIDER_KEY>",
"reasoningEffort": "low"
}
}
Header: Authorization: Bearer <YOUR_CHUNKCHEF_KEY>.
| Request field | What it is | ”Variable” |
|---|---|---|
Authorization | your ChunkChef API key | API access key |
sourceUrls[] | file URLs | document links |
prompts[0] | the entire template as a single string | structured prompt |
neural.type / neural.model | provider and model | model |
neural.apiKey | provider key (BYOK) | model key |
neural.reasoningEffort | optional minimal/low/medium/high | reasoning depth |
ocr.provider | OCR provider enum (e.g. NEURAL_CLIENT_TYPE_MISTRAL) | OCR provider |
ocr.model | OCR model identifier; required | OCR model |
ocr.providerKey | Provider key for OCR (BYOK); accepted as input only, never returned | OCR key |
extractionMode | Text-extraction strategy: EXTRACTION_MODE_HYBRID (default, “Optimized Hybrid Extraction”) lets the converter decide text-extract vs OCR per file; EXTRACTION_MODE_OCR_ALWAYS forces neural OCR on every file; omitted/EXTRACTION_MODE_UNSPECIFIED inherits your account default | Optional |
responseSchema | Optional JSON Schema (raw string) validating the model’s JSON answer; self-contained, strict-subset-compatible (see “Structured output”) | Optional |
More examples (briefly). Same mechanics — only the prompt text and the expected answer shape in llm[].content differ:
- Classification. Prompt: “Determine the document type: invoice / contract / receipt / letter / other. Return a single word from the list, with no explanation.” Answer: a single word (e.g.
contract). - Contract terms. Prompt: “Extract: parties, subject, amount, term, and termination conditions. Return JSON matching the schema {parties[], subject, amount, term, termination}. Field not found → null.” Answer: JSON matching the schema.
- Summary. Prompt: “Summarize the document in 3–5 sentences. No bullet lists.” Answer: prose text.
Build your own prompt (meta-prompt)
The fastest way to get a high-quality prompt is to have a capable model write it for you. Hand it the meta-prompt below: paste in our example as the reference structure, the output shape you need (JSON/CSV/Markdown), and a description of your context and task — and you get a ready-to-use ChunkChef prompt back. Fill in the bracketed blocks; the model handles the rest.
You are a senior prompt engineer. Build a production-grade extraction prompt that will be sent to a document-processing model through the ChunkChef API. The model receives the OCR'd text (HTML) of a single document and must return data in a strict, machine-parseable format.
WHAT I'M GIVING YOU
1) REFERENCE PROMPT — the structure and style to follow. Preserve its section anatomy.
<<<REFERENCE_PROMPT
[paste the ChunkChef example prompt here]
REFERENCE_PROMPT
2) TARGET OUTPUT — the exact shape I need back: a JSON schema/sample, CSV columns, or Markdown layout.
<<<TARGET_OUTPUT
[paste your desired JSON / CSV / Markdown here]
TARGET_OUTPUT
3) DOMAIN & CONTEXT — what these documents are, where they come from, and their quirks (languages, layouts, stamps, tables, common OCR errors).
<<<CONTEXT
[describe your documents and domain]
CONTEXT
4) TASK — exactly what to extract or produce, plus the business rules, definitions, and edge cases that matter.
<<<TASK
[describe the task and rules]
TASK
5) OUTPUT FORMAT — one of: JSON | CSV | Markdown. Default: JSON.
<<<FORMAT
JSON
FORMAT
HOW TO BUILD THE PROMPT
1. Study the domain and task deeply before writing. Infer the edge cases a careful human reviewer would catch — ambiguous fields, duplicates, ranges, units, missing data, multi-row tables, appendices — and address each one explicitly.
2. Keep the REFERENCE PROMPT's anatomy: a one-line TASK, an IMPORTANT "only the target format" rule, then numbered sections (INPUT, OUTPUT FORMAT, EXTRACTION/PROCESSING RULES field by field, PROCESSING REQUIREMENTS, RESPONSE FORMAT), and a final REMEMBER reinforcement.
3. Make the output contract unambiguous for the chosen format:
- JSON: give the full schema with types and nullability, mark required vs optional keys, forbid any text/markdown/code fences outside the JSON, and require the answer to start with "{" (or "[") and end with "}" (or "]").
- CSV: fix the exact column order and header row, the delimiter, the quoting/escaping rule, and how empty values are written; one record per row, no prose.
- Markdown: fix the exact headings/table columns and forbid any content outside that layout.
4. Pin the data discipline: use only the provided document text; never invent, guess, or reformat beyond the normalization you explicitly define; specify number, date, and unit normalization; define how "not found" is represented (null / empty / skipped) and how duplicates are handled; preserve source values verbatim where identity matters.
5. Account for ChunkChef specifics: the model sees one document's OCR'd HTML, possibly truncated or split into chunks; do not rely on any temperature setting — enforce determinism through wording ("be literal and deterministic"); the prompt is sent verbatim, so resolve every "<placeholder>" yourself.
6. Self-check before finishing: re-read the TARGET OUTPUT and confirm the prompt forces exactly that shape, that every field has a rule, and that a small, cheap model could follow it without guessing.
OUTPUT
Return ONLY the finished prompt, ready to paste into ChunkChef's "prompts" array — no explanation, no preamble, no code fences.
Connecting a model (BYOK)
The LLM stage runs on your provider key. The configuration is passed in the neural object when you create a job:
| Field | Required | Description |
|---|---|---|
type | yes | provider (see list below) |
model | yes | model identifier; passed to the provider as-is |
apiKey | yes | your provider key; accepted as input only, never returned in responses |
reasoningEffort | no | reasoning-depth hint; allowed values: minimal, low, medium, high (empty = off). An invalid value → 400 error. Whether it’s honored depends on the provider/model. |
chunkBudgetTokens | no | token budget for a single model call: covers both the prompt and the document text in one chunk. 0/unset → the service’s conservative default. Range: 16000–2000000; a value out of range → 400. Set it to your model’s context window — only you know it. The response always returns the actual effective budget (including when you rely on the default): the service reserves a small allowance for chat-template tokens, so the returned value is slightly lower than what you set. |
Supported providers:
| Provider | neural.type value |
|---|---|
| OpenAI | NEURAL_CLIENT_TYPE_OPENAI |
| Anthropic (Claude) | NEURAL_CLIENT_TYPE_CLAUDE |
| xAI (Grok) | NEURAL_CLIENT_TYPE_GROK |
| Together | NEURAL_CLIENT_TYPE_TOGETHER |
| DeepSeek | NEURAL_CLIENT_TYPE_DEEPSEEK |
| Xiaomi | NEURAL_CLIENT_TYPE_XIAOMI |
| Mistral | NEURAL_CLIENT_TYPE_MISTRAL |
| OpenRouter | NEURAL_CLIENT_TYPE_OPENROUTER |
Through NEURAL_CLIENT_TYPE_OPENROUTER you get models from many vendors that don’t have a direct integration.
You pay the provider directly at their rate — ChunkChef adds no markup on tokens or OCR.
Extraction mode (Optimized Hybrid Extraction)
By default (EXTRACTION_MODE_HYBRID), the converter picks text-extract vs OCR per file — the “Optimized Hybrid Extraction” strategy. Set extractionMode to EXTRACTION_MODE_OCR_ALWAYS on a job to force neural OCR on every file, regardless of whether it already has an extractable text layer. Note: this means every file becomes a provider OCR call — a cost increase versus hybrid mode. A job’s own extractionMode overrides your account default, which you set in the dashboard under Settings.
Structured output (responseSchema)
Pass an optional responseSchema — a JSON Schema, as a raw JSON string — on job creation to have ChunkChef validate the shape of the model’s JSON answer, in addition to (not instead of) describing that shape in your prompt.
The schema must be self-contained: only internal #/$defs references are allowed — no external $ref, and no remote $schema/$id. Limits: at most 128 KiB, nesting depth 64, 10000 nodes total. An invalid or out-of-limits schema is rejected with 400 at job creation, before any file is processed. For OpenAI and Grok, structured outputs run natively and require ChunkChef’s strict subset: every property listed in required (express optionality as type: [..., "null"], not omission) and additionalProperties: false on every object. Writing your schema to that strict subset makes it portable — the same schema works unchanged across every supported provider. A schema that passes ChunkChef’s create-time checks but isn’t strict-subset-compatible for OpenAI/Grok doesn’t fail at creation and never reaches the schemaValid/schemaErrors flow — instead, that row’s llm[] entry ends as JOB_LLM_STATUS_FAILED with a provider error, since strict-subset conformance is enforced by the provider, not by ChunkChef’s create-time check.
Two fields on each llm[] row report the outcome (see “API reference”):
schemaValid— whether the row’scontentconforms toresponseSchema. Present only when the file wasn’t split into chunks (chunkTotal= 1); omitted for multi-chunk files — the same schema is still sent with every chunk’s call, but per-chunk verdicts aren’t rolled up into a whole-document one yet.schemaErrors— the conformance violations whenschemaValidisfalse(up to 10, each up to 512 bytes).
If any row’s answer doesn’t conform, the job finishes as JOB_STATUS_PARTIAL instead of JOB_STATUS_COMPLETE — content is still returned as-is, just flagged. For providers ChunkChef emulates structured outputs for (DeepSeek, Xiaomi), a non-conforming first answer gets one automatic repair retry before it’s scored.
Merging chunked results (merge)
When a document is split into chunks (see “Prompts and data extraction” above), each chunk gets its own model call and its own llm[] row — combining those pieces into one answer is, by default, up to you. Opt into merge on job creation and ChunkChef does that combining for you, on the server: the result gains a merged[] array holding one whole-document (or whole-file) answer, built from the per-chunk answers.
When to enable it. If you’re extracting structured JSON from a document long enough to be split into several chunks — say, a multi-page invoice whose line items span page 1 and page 2 — merge saves you from writing the reassembly code yourself: ChunkChef unions the JSON fields, concatenates and deduplicates arrays, and resolves any field that disagrees between chunks.
Turn it on:
{ "merge": { "enabled": true } }
Every other merge.* field is optional and takes a safe default (see “Merge options” in the API reference for the full list).
Example: a two-page invoice
Picture a 2-page invoice, OCR’d into text long enough to be split into 2 chunks — page 1 lands in chunk 0, page 2 in chunk 1. Your prompt asks for JSON: { "supplier": "...", "items": [...], "total": "..." }. Without merge, you get two separate llm[] answers, one per chunk, and combine them yourself. With merge on, ChunkChef combines them into a single JSON object: the items arrays from both chunks are concatenated into one, and supplier/total are taken from whichever chunk actually reports them. If chunk 0 and chunk 1 each report a different total for the same document (say the OCR duplicated a subtotal line, or the model misread a number), that’s a genuine conflict — see “Resolving conflicting values” below.
Scope: one answer per file, or one across the whole job (merge.scope)
merge.scope controls how much gets combined into a single merged[] entry:
MERGE_SCOPE_FILE(default) — merge chunks within each file separately: onemerged[]entry per file, withmerged[].fileset to that file’s URL.MERGE_SCOPE_JOB— merge chunks across all files of the job into a single answer: onemerged[]entry, withmerged[].fileempty (""). Use this when the files you uploaded are pages or parts of one logical document (e.g. a purchase order split across several source files) and you want one combined result instead of one per file.
Resolving conflicting values (merge.conflictPolicy)
When merging JSON, two chunks can disagree about the same field — one reports total: 1500, another total: 1520 for the same invoice. merge.conflictPolicy decides which value wins:
MERGE_CONFLICT_POLICY_FIRST_NON_NULL(default) — the first non-null value, in chunk order, wins.MERGE_CONFLICT_POLICY_MAJORITY— the value that appears most often wins; with fewer than 3 chunks, or when no value has a strict majority, ChunkChef falls back toFIRST_NON_NULL.
Either way, every such disagreement is recorded in merged[].conflicts[] — the field, the competing values, and which file/chunk each came from — so a mismatched total isn’t silently papered over. Check conflicts[] when you need to know whether the winning value is trustworthy or a human should take a look.
Two kinds of deduplication
Merge removes duplicate content in two different ways:
- Technical deduplication — always on, no configuration needed. Chunks of the same file typically overlap a little at their boundary (the same table row appears at the end of one chunk and the start of the next); merge detects and removes that overlap automatically. This is the safe default — it never removes content beyond that boundary overlap.
- Content deduplication — opt-in, via
merge.dedupeBy. Atscope=job, the same logical record can legitimately appear in more than one file (e.g. a line item repeated across two related documents), and that repeat might be intentional — so ChunkChef leaves it alone unless you ask. Setmerge.dedupeByto the JSON field names that identify a unique record (e.g.["invoiceNumber", "lineNo"]) and ChunkChef collapses records that match on those fields, keeping the first occurrence. Leavemerge.dedupeByempty — the default — to keep only technical dedup, nothing more.
merge.dedupeBy works with JSON output only, and requires responseSchema to be set — the field names are looked up as plain keys on the parsed JSON object. At most 32 field names are allowed; more are rejected with a 400 at job creation.
Both kinds of removal are counted separately in the result, so you can tell them apart:
merged[].dedupeRemovedTechnical— records removed as automatic boundary-overlap dedup.merged[].dedupeRemovedContent— records removed by yourdedupeByfields.
Reading the result: merged[] vs raw llm[]
When merge is on, the result gains merged[] alongside the existing llm[]:
merged[]— the server-combined, whole-document (or whole-file) answer. Read this as your primary result.llm[]— unchanged: still one row per chunk. It stays available so you can inspect exactly what each chunk’s model call returned — useful when debugging a conflict.
If you set responseSchema, the merged JSON object is validated against it and merged[].schemaValid reports the outcome — the same way llm[].schemaValid does for a single-chunk file today.
When a chunk couldn’t be merged (merged[].mergeIncomplete)
Merging is best-effort: if one chunk’s answer doesn’t parse, or a document structure spans chunks in a way merge can’t safely combine, that chunk’s raw content is still included — appended, not merged — and merged[].mergeIncomplete is set to true, with the affected chunk(s) listed in merged[].incompleteChunks. The rest of the merge still completes normally; check this flag when you need to know whether the merged result is fully clean or partially assembled.
Consensus runs (consensus)
For a schema job (responseSchema set), you can have ChunkChef run the same chunk through the model multiple times and vote on the answer field-by-field, instead of trusting a single run. Opt in with consensus.mode on job creation: ChunkChef runs the chunk k times, takes the majority (plurality) value for each field, and reports both the agreed answer and every field it wavered on — a confidence signal on top of your single BYOK call, at k× the token cost on your own key. Because the runs are sequential, consensus also multiplies processing time — a job can take up to k× longer.
Requires responseSchema. Consensus needs a schema to know what a “field” is — without one there’s nothing to vote on. Setting consensus.mode without responseSchema is rejected with 400 at job creation.
Turn it on:
{ "responseSchema": "...", "consensus": { "mode": "CONSENSUS_MODE_THREE_RUNS" } }
Modes
The mode name carries the token multiplier — there’s no separate “runs” count to set:
| Mode | Runs | Notes |
|---|---|---|
CONSENSUS_MODE_TWO_RUNS | 2 | cheapest — an instability probe, not a reliable vote (see caveats below) |
CONSENSUS_MODE_THREE_RUNS | 3 | recommended default — a real plurality vote |
CONSENSUS_MODE_FIVE_RUNS | 5 | maximum scrutiny, highest cost |
Omitting consensus (or CONSENSUS_MODE_UNSPECIFIED) disables it — behavior is byte-identical to a job without consensus.
Reading the result: minAgreement and disagreements
Each llm[] row gains a consensus object (see “API reference” for the full field list). The two things to read first:
minAgreement— the lowest per-field agreement across the row, as a fraction (1.0= every voted field agreed on every run). A value of0is a reserved sentinel meaning no real vote happened (fewer than 2 votable runs, or the vote degraded) — read it as “consensus unavailable for this chunk,” not as “0% agreement.”disagreements[]— one entry per field (or array) the runs didn’t fully agree on, worst-agreement first. Each entry lists the competingvariants[]: the value, how many runs produced it, and whether itincluded— won the vote and is present in the agreed answer. A variant can haveincluded: trueeven when its value isnull: a winningnullmeans the field’s key is omitted from the agreed JSON, not that a literalnullappears.
Honest caveats
CONSENSUS_MODE_TWO_RUNSis a probe, not a vote. Atk=2a scalar disagreement always splits 1-vs-1 (agreement0.5), and on arrays nothing is ever filtered — every element either run reported ends up in the agreed answer. Two runs tell you that the model wavered; only three runs or more actually filter out a minority value.- The plurality vote can drop an entity every run agreed exists. If an array element is present in every run but with one differing cell (e.g. a line item every run extracted, but one run misread a single number), each run’s version counts as a distinct canonical value — at
k≥3each gets 1-of-k, below the majority threshold, and the element can be dropped from the agreed array even though every run agreed it belongs there.disagreements[]still exposes every variant, so the raw evidence survives even when the agreed answer doesn’t include it — check it when an array looks shorter than expected. - An oversized answer counts as unvotable, not as an error. A run’s answer that parses as valid JSON but exceeds ChunkChef’s internal parse limits is excluded from the vote; the vote proceeds over the remaining runs — the same treatment as a run that returns non-JSON text.
- Consensus and
mergecompose as two independent votes. With both enabled, each chunk is first voted on internally (kruns → one agreed per-chunk answer), and then, ifmerge.conflictPolicyisMERGE_CONFLICT_POLICY_MAJORITY, the merge step votes again across chunks’ agreed answers. Nothing carries over between the two votes — a field the per-chunk consensus already resolved is presented to merge as a settled value, like any other chunk’s answer.
Idempotent retries
To retry job creation safely, generate one idempotencyKey (a UUID) and reuse it
across retries of the same request:
POST /v1/jobs
{ "sourceUrls": ["..."], "ocr": { ... }, "neural": { ... }, "idempotencyKey": "3f1c…" }
Retrying with the same key and identical parameters returns the original job;
changing any parameter under the same key returns 400. Use a new key for a
genuinely new job.
Webhooks
Webhooks let you avoid polling: the service will POST your endpoint once the job finishes. This is entirely optional — if you don’t set the fields, API behavior is unchanged.
Setup
When creating a job (POST /v1/jobs), pass one or both optional fields:
| Field | Type | Description |
|---|---|---|
webhookUrl | string | Absolute URL of your endpoint (http:// or https://). Accepted as input only — never returned in responses. |
webhookSecret | string | Optional HMAC signing secret. Accepted as input only — never returned in responses; stored encrypted at rest and deleted with the job (see “Security and data”). |
Example:
{
"sourceUrls": ["https://example.com/invoice.pdf"],
"prompts": ["Extract the total."],
"ocr": { "provider": "NEURAL_CLIENT_TYPE_MISTRAL", "model": "mistral-ocr-latest", "providerKey": "..." },
"neural": { "type": "NEURAL_CLIENT_TYPE_XIAOMI", "model": "mimo-v2-flash", "apiKey": "..." },
"webhookUrl": "https://your-service.example.com/hooks/ChunkChef",
"webhookSecret": "my-secret-value"
}
When the webhook fires
Once per job — on the first transition to a terminal status (JOB_STATUS_COMPLETE, JOB_STATUS_PARTIAL, or JOB_STATUS_FAILED). Delivery itself is at-least-once (duplicates are possible on failure); see “Delivery guarantees.” The webhook doesn’t carry the processing result; it signals completion only. Fetch the full result with the usual GET /v1/jobs/{id}/result.
Request body
The service sends a POST to your webhookUrl with Content-Type: application/json and a JSON body:
{
"job_id": "15b07304-...",
"account_id": "a1b2c3d4-...",
"status": "complete",
"finished_at": "2026-06-24T12:34:56Z"
}
| Field | Type | Description |
|---|---|---|
job_id | string | Job UUID |
account_id | string (account identifier) | Account identifier |
status | string | One of: complete, partial, failed |
finished_at | string | Job completion time in RFC3339 format (UTC) |
Signature verification
When webhookSecret is set, every request includes two additional headers:
| Header | Example value | Description |
|---|---|---|
X-Chunkchef-Timestamp | 1750765200 | Unix time of delivery (seconds) |
X-Chunkchef-Signature | sha256=a3f4... | HMAC-SHA256 signature |
Signing algorithm:
signature = "sha256=" + hex( hmac_sha256(secret, "<timestamp>.<body>") )
where <timestamp> is the string form of the Unix time from X-Chunkchef-Timestamp, <body> is the raw request body (bytes as received), and . is the separator. hex is lowercase; secret is used as UTF-8 bytes.
How to verify on your side:
- Extract the
X-Chunkchef-Timestampvalue. - Compute
hmac_sha256(secret, "<X-Chunkchef-Timestamp value from step 1>.<raw request body>"), encode as lowercase hex, and prependsha256=. - Compare it to
X-Chunkchef-Signatureusing a constant-time comparison (hmac.Equal/crypto/subtle.ConstantTimeCompareor equivalent). - Reject the request if the timestamp is too old (recommended tolerance: 5 minutes).
If webhookSecret is not set, the X-Chunkchef-Timestamp and X-Chunkchef-Signature headers are not sent.
Retry policy
If your endpoint is unreachable or returns an error, the service retries delivery on the following schedule:
| Attempt | Delay before next |
|---|---|
| 1 → 2 | 1 minute |
| 2 → 3 | 5 minutes |
| 3 → 4 | 15 minutes |
| 4 → 5 | 30 minutes |
| 5 → 6 | 30 minutes |
| 6 | — (final; delivery is marked failed afterwards) |
Total: up to 6 attempts.
Permanent errors (4xx other than 408/429, unusable URL, SSRF block) are not retried: the delivery is immediately marked as failed. A 2xx response is considered a success.
Delivery guarantees
Delivery is at-least-once: in most cases your endpoint receives exactly one call, but retries can cause duplicate delivery on failures. Deduplicate events on your side using job_id.