True Streaming CAR Uploads

[rvagg]

• Follow-up to feat: support streaming uploads #287
• Related to Support large files in filecoin-pin #190

Problem

Current CAR building buffers entire content before upload:

• Node.js: Writes CAR to temp file, then streams file to Synapse
• Browser: Accumulates CAR bytes in memory (up to 1GiB), then uploads

Two reasons for this:

1. CAR header patching: We write a placeholder root CID in the CAR header, build the DAG, then patch/rewrite the header with the actual root. This requires the CAR to be fully built before streaming. We can eliminate this by using ZERO_CID in the header instead (valid CAR, signals "no meaningful root"; technically an empty roots array is fine but there are some historical compatibility issues that make having one the safe option).

2. Synapse metadata timing: The upload() API requires IPFS root CID metadata at call time, but root CID is only known after DAG building completes. This requires API changes in Synapse.

Blocker

Synapse SDK's upload() API requires metadata at call time. The IPFS root CID (ipfsRootCID metadata key) must be provided when calling upload(), but we don't know it until DAG building completes.

Blocked by: FilOzone/synapse-sdk#494 - Split upload() into store() + commit()

Solution

Once Synapse provides separate store() and commit() operations:

1. CAR builder changes:

   • Use CarWriter.create([ZERO_CID]) (where ZERO_CID = bafkqaaa)
   • Return { carStream: ReadableStream, rootCidPromise: Promise<CID> } instead of buffered bytes/file
   • Remove updateRootCidInCar() post-processing

2. Upload flow changes:

   • Call store(carStream) - streams bytes directly as DAG builds
   • Await rootCidPromise - resolves when UnixFS completes
   • Call commit(pieceCid, { ics: rootCid }) - registers with metadata

3. Files to modify:

   • src/core/car/car-*-backend.ts - expose out stream, use ZERO_CID
   • src/core/unixfs/car-builder.ts - return streaming result
   • src/core/unixfs/browser-car-builder.ts - return streaming result
   • src/core/upload/synapse.ts - use new store/commit API
   • src/add/add.ts, src/import/import.ts, src/filecoin-pin-store.ts - consume streaming API

[rvagg]

This is going to need a little bit of e2e testing to ensure that the ZERO_CID isn't a problem anywhere in the stack. It shouldn't be, and I'm also tempted to just do an empty roots array instead and test that out but the conservative version is ZERO_CID.