Publish large models (connectors)

Upload large host-application models using SendPipeline and Model Ingestion — connector authors only

When a desktop connector converts an entire host-application model, Send and Send2 cannot hold the full graph in memory. SendPipeline streams to a local pack, uploads once, and tracks progress through client.Ingestion.

When to read this: production desktop connectors only — scripts use Automate with scripts. Read next: Load and publish model data Path E.

When to Skip This Page

You are…	Use instead
Writing a script, notebook, or Grasshopper C# component	Automate with scripts with Send2/Receive2
Following the Quickstart or sending a small graph	Send/Receive or Send2 — see Load and publish model data
Building a production desktop connector for a host app	This page

Connector-only pathway. This guide is for production desktop connectors uploading large host-application models. If you are writing a script, Grasshopper C# component, notebook cell, or service — not converting a whole host-application model — use Automate with scripts with Send2/Receive2 instead. See Load and publish model data for the decision guide.

Prerequisites

• Speckle Server version 3.0.3 or later (client.Server.IsWorkspaceEnabled() doesn’t gate this). On server 3.0.11+, call client.Model.CanCreateModelIngestion(project.id, model.id) to verify availability; on 3.0.3–3.0.10, confirm with your server admin.
• Authentication and a project and model already created
• Familiarity with Load and publish model data

Detecting whether a server supports model ingestion at all, and telling that apart from a user only lacking permission, is exactly the pattern covered in Handling server capabilities — read it before writing the availability check below for anything beyond a one-off script.

Complete Example

Complete example

using Microsoft.Extensions.DependencyInjection;
using Speckle.Sdk.Api.GraphQL.Inputs;
using Speckle.Sdk.Models.Collections;
using Speckle.Sdk.Pipelines.Progress;
using Speckle.Sdk.Pipelines.Send;

// 1. Create the ingestion job
var sourceData = new SourceDataInput("my-connector", "1.0.0", fileName: null, fileSizeBytes: null);
var ingestion = await client.Ingestion.Create(
    new ModelIngestionCreateInput(model.id, project.id, "Publishing model", sourceData)
);

// 2. Create the pipeline and stream objects into it one at a time as you build the graph
using var pipeline = provider.GetRequiredService<ISendPipelineFactory>()
    .CreateInstance(ingestion, client.Account, new Progress<StreamProgressArgs>(), cancellationToken: default);

var root = new Collection { name = "My model" };

try
{
    foreach (var element in myLargeModelElements)
    {
        Base converted = ConvertToSpeckle(element);
        // Process() returns this object's own reference — embed it in the parent
        // instead of holding the fully-converted object graph in memory.
        ObjectReference reference = await pipeline.Process(converted);
        root.elements.Add(reference);
    }

    // The root is processed last, referencing all the children already on disk
    await pipeline.Process(root);
    await pipeline.WaitForUpload();

    // WaitForUpload() itself tells the server the pack is ready to process — the
    // server creates the version asynchronously once processing succeeds. Subscribe
    // to get notified when it's ready (see "Finding Out When the Version Is Ready" below).
}
catch (Exception ex)
{
    await client.Ingestion.FailWithError(ModelIngestionFailedInput.FromException(ingestion.id, project.id, ex));
    throw;
}

This example follows the streaming pattern used by every Speckle desktop connector — see speckle-sharp-connectors, RhinoContinuousTraversalBuilder and SendOperation.SendViaPackfile, for a full production implementation.

Step-by-Step Explanation

1. Create the Ingestion Job

An ingestion is the server-side tracking record for this upload:

Example

var sourceData = new SourceDataInput("my-connector", "1.0.0", fileName: null, fileSizeBytes: null);
var ingestion = await client.Ingestion.Create(
    new ModelIngestionCreateInput(model.id, project.id, "Publishing model", sourceData)
);

SourceDataInput records which application and version produced this data — useful for diagnostics on models that fail to process.

2. Process Objects One at a Time, Not the Whole Graph

ISendPipelineFactory.CreateInstance builds a pipeline bound to this specific ingestion:

Example

using var pipeline = provider.GetRequiredService<ISendPipelineFactory>()
    .CreateInstance(ingestion, client.Account, new Progress<StreamProgressArgs>(), cancellationToken: default);

Process does recursively serialize an entire object graph in one call — including nested Base values marked [DetachProperty] or prefixed with @ — and writes every resulting object (root and detached children) to the local, gzip-compressed NDJSON file. The reason production connectors still call Process bottom-up, once per converted element, and substitute a lightweight ObjectReference in the parent instead of keeping the full child object, is memory, not a limitation of Process itself: converting an entire host-application model into one in-memory Base graph before calling Process would require holding every converted mesh and property in memory at once. Processing and writing each object as soon as it’s converted lets you drop it from memory immediately afterward.

Example

ObjectReference reference = await pipeline.Process(convertedObject);
parentCollection.elements.Add(reference); // parent references the child by ObjectReference, not the object itself
// ...
await pipeline.Process(root); // root, last, once every child is on disk
await pipeline.WaitForUpload();

Process returns immediately once the object is written to the local temp file — it does not touch the network. WaitForUpload is what actually uploads: it finalizes the local file, uploads it to storage, and tells the server to start processing it. Always call both, and always call WaitForUpload even if you don’t care about the return value — skipping it means nothing is ever sent. SendPipeline implements IDisposable — always wrap it in a using.

uploadProgress is a required IProgress<StreamProgressArgs> — pass new Progress<StreamProgressArgs>() if you don’t need to report progress to a UI, matching the pattern used by Send/Send2.

3. Finding Out When the Version Is Ready

Unlike Send2 + Version.Create, this pathway doesn’t hand you a version id synchronously. WaitForUpload only confirms the upload reached the server and processing has started — the server decompresses the pack, materializes the objects, and creates the version asynchronously, after your call returns.

Production connectors do not call client.Ingestion.Complete after WaitForUpload on this pathway — the server completes the ingestion and creates the version itself once processing finishes. Calling Complete is for a different pattern: tracking a classic Send2 call with an ingestion record you complete manually (see ModelIngestionResource). Mixing the two — sending via SendPipeline and then also calling Complete — is redundant and not how any current Speckle connector uses this API.

To learn when the version is ready, subscribe to ingestion updates and read versionId off the success status:

Example

using var subscription = client.Subscription.CreateProjectModelIngestionUpdatedSubscription(
    new ProjectModelIngestionSubscriptionInput(
        project.id,
        new ModelIngestionReference(ingestion.id, null),
        ProjectModelIngestionUpdatedMessageType.updated
    )
);

subscription.Listeners += (_, update) =>
{
    var status = update.modelIngestion.statusData;
    if (status.status == ModelIngestionStatus.success)
    {
        Console.WriteLine($"Version ready: {status.versionId}");
    }
};

If your application can’t hold a subscription open, poll client.Ingestion.Get(ingestion.id, project.id) instead and inspect its status.

If anything goes wrong on the client before WaitForUpload returns, report the failure so the ingestion doesn’t stay stuck in processing. ModelIngestionFailedInput.FromException builds the input from a caught exception:

Example

await client.Ingestion.FailWithError(ModelIngestionFailedInput.FromException(ingestion.id, project.id, ex));

Use FailWithError for unexpected/infrastructure failures, FailWithInvalid when the input itself was invalid (unsupported file, bad settings), and FailWithCancel only when the user explicitly cancelled. See ModelIngestionResource for the full state machine.

Cooperative Cancellation

If your connector supports cancelling an in-progress send, don’t cancel the pipeline unilaterally — request cancellation, then let the sending client observe and report it:

Example

// Caller / UI side: request cancellation
await client.Ingestion.RequestCancellation(
    new ModelIngestionCancelledInput(ingestion.id, project.id, "User cancelled")
);

// Sending client: observe the request (via subscription) and report cooperatively
await client.Ingestion.FailWithCancel(
    new ModelIngestionCancelledInput(ingestion.id, project.id, "User cancelled")
);

RequestCancellation does not stop the pipeline by itself — it’s a signal the actively-sending process must poll or subscribe to and act on.

Common Pitfalls

Ingestion stuck in 'processing' forever

On the client side, you only need to reach a terminal call if something goes wrong before or during upload: FailWithError, FailWithInvalid, or FailWithCancel. If WaitForUpload succeeds, the server takes over and drives the ingestion to success or failed itself — you don’t call Complete. If an ingestion is stuck in processing with no client-side error, that’s a server-side processing problem (corrupt pack, unsupported data), not a missing client call — inspect it with client.Ingestion.Get.

Calling client.Ingestion.Complete after WaitForUpload

This is unnecessary for the SendPipeline pathway and isn’t how any current Speckle connector uses this API — see Finding Out When the Version Is Ready above. Complete exists for a different, manual ingestion-tracking pattern built around Send2.

Building the whole model in memory before calling Process() once

Process does correctly serialize and write an entire nested object graph — including detached children — from a single call at the root. The reason to avoid this pattern at connector scale isn’t correctness, it’s memory: converting every element of a large host-application model before calling Process means holding the whole converted graph in memory at once. Convert and Process bottom-up instead, replacing each processed child with its ObjectReference in the parent, so converted objects can be released from memory as soon as they’re written to disk.

Object IDs don't match IDs from Operations.Send

SendPipeline uses a different serializer than Send/Send2. Don’t compare or reuse IDs across the two pathways — see Load and publish model data.

404 or unsupported errors calling client.Ingestion methods

The Model Ingestion API requires server version 3.0.3 or later. On 3.0.11+, use client.Model.CanCreateModelIngestion(project.id, model.id) to verify availability before enabling this pathway; on 3.0.3–3.0.10, confirm compatibility with your server admin. Don’t wrap the whole pathway in a broad try/catch to detect this — see Handling server capabilities for the narrow-catch pattern that tells “server too old” apart from “not authorized.”

Next Steps

Load and publish model data

Path E decision guide

Handling server capabilities

Permission and version checks

ModelIngestionResource

Ingestion lifecycle API