For a full list of available versions, see the Directory of published versions

4.1.3. Bulk Load of Immunization Document Bundles

The Bulk Load interaction is used to migrate historical immunization records to the IR. This encompasses both Create and Update.

4.1.3.1. Overview

Note: This page is a first draft covering the bulk data migration ("bulk load") pathway for onboarding historical bulk extracts of immunization data, beginning with Panorama and extensible to other source systems (e.g. pharmacies, EMRs).

The Bulk Load interaction is used to migrate immunization records into the IR in volume, via file-based exchange, as distinct from the transactional, per-record Submit Immunization interaction. This pathway is intended for initial data migration from source systems (e.g. Panorama) and may also be used for periodic bulk extracts from other source systems going forward.

4.1.3.2. Bulk Load Prerequisite

• Patient data from the source system gets submitted to the Provincial Client Registry (PCR).
• MRNs of all patients in the files prepared for Bulk Load get validated offline prior to the start of the Bulk Load processing to ensure that these MRNs exist in PCR
  • No additional PCR validation is done during the Bulk Load

4.1.3.3. Usage

A Source System exports its immunization data as one or more NDJSON (newline-delimited JSON) files and uploads them to a designated location (location details will be determined by Ontario Health and communicated to the implementers at onboarding time). Each line of an NDJSON file is a single FHIR Bundle of type document, structured identically to the Bundle used in the Submit Immunization interaction (Composition as the first entry, with Composition.section.entry referencing an Immunization resource, plus the associated Patient, MedicationRequest, PractitionerRole, Practitioner, Organization, and Location resources). This Bundle model constitutes a self-contained immutable point-in-time document.

IR Bulk Load picks up the uploaded NDJSON file(s), reads each line, and ingests it into the IR Repository using a conditional update based on the Bundle's client-assigned (i.e. source system assigned) business identifier to handle both initial creation and subsequent re-submission of corrected records without creating duplicates.

During ingestion, each record is validated against the IR profiles. Records that pass validation are stored, and a Provenance resource is created for every ingested record to capture the origin of the data and its source system. Records that fail validation, along with their associated OperationOutcome resources, are routed to Data Quality Solution (DQS). DQS compiles error reports, and IR Bulk Load team works with the source system to correct the underlying data; the source system then re-extracts the corrected records into a new NDJSON file for re-submission through the same pathway.

Actor: Source System
Role: Extracts immunization data and exports it as NDJSON file(s). Each line is a document Bundle with a stable, client-assigned identifier suitable for conditional update. Uploads file(s) to the designated location, and re-extracts and re-submits corrected records identified by DQS.

Actor: Upload Location
Role: Intermediate storage location on Azure Blob Storage where Source Systems deposit NDJSON files for retrieval by the IR Bulk Load. Mechanism, protocol, and ownership will be determined by Ontario Health and communicated to the implementers at onboarding time.

Actor: IR Bulk Load
Role: Retrieves uploaded NDJSON files, parses each line as a document Bundle, converts it into a transaction Bundle entry using a conditional update keyed on the Bundle's client-assigned identifier, and submits it to the IR Solution. Aggregates per-file processing results and communicates this batch summary along with the error report and rejected records to the Source System to resolve data issues ahead of re-submission.

Actor: IR Solution (Smile CDR)
Role: Validates each submitted record against the IR profiles, performs patient matching, creates or updates the immunization record, creates a Provenance resource for every ingested record, and returns an OperationOutcome with errors and warnings for any record that fails validation or business rules.

Actor: Data Quality Solution (DQS)
Role: Receives rejected records and OperationOutcome resources from the IR Bulk Load, compiles error reports, and forwards these to IR Bulk Load.

4.1.3.4. Specification

4.1.3.4.1. File format

• Files are NDJSON (newline-delimited JSON): one FHIR Bundle per line, no enclosing JSON array.
• Each line is a Bundle of type = document, with Bundle.entry[0] having a Composition resource, and Composition.section.entry referencing an Immunization resource (plus all the associated Patient, MedicationRequest, PractitionerRole, Practitioner, Organization, and Location resources) - structurally identical to the Bundle used for the Submit Immunization interaction.
  • This Document Bundle model constitutes a self-contained immutable point-in-time document.
• Maximum file size: 10 MB per file. Source Systems must split larger extracts across multiple files.
• NDJSON file naming convention: PanBulkLoad_YYYYMMDD_FileNNNNNNN.ndjson (e.g. PanBulkLoad_20260810_File0000001.ndjson).
• Upload location on Azure Blob Storage / protocol will be determined by Ontario Health and communicated to the implementers at onboarding time.
• NDJSON files are processed independently of each other - order of the files is not important, and each file is processed regardless of the success or failure of processing of other files
• Lines in the NDJSON files are processed independently of each other - order of the lines is not important, and each line is processed regardless of the success or failure of processing of other lines
• Character encoding: UTF-8.

4.1.3.4.2. Client-assigned identifier

Every Bundle SHALL have a stable, client-assigned identifier that the Source System will not reuse for a different record. This is typically:

• An existing unique immunization identifier from the source system, or
• A primary key field from the source system's database table, if no immunization-specific identifier exists. This identifier is used as the conditional update search parameter so that re-processing the same record (e.g. after a correction) updates the existing IR record rather than creating a duplicate.

4.1.3.4.3. Provenance

For every record successfully created or updated through this pathway, the IR Solution creates a Provenance resource recording:

• The source system as Provenance.agent.
• The bulk load activity/mechanism (to distinguish bulk-migrated records from records submitted via the transactional Submit Immunization interaction).
• The date/time of ingestion.

4.1.3.4.4. Error handling and re-submission

• Records that fail structural, profile, or business-rule validation are not stored. The corresponding OperationOutcome, together with the rejected record, is forwarded to DQS.
• IR Bulk Load reports errors (including file name, line number and record identifier) and rejected records back to the Source System.
• The Source System corrects the underlying data and re-extracts affected records into a new NDJSON file, which is processed through the same pathway. Because processing is keyed on the client-assigned identifier, a corrected record replaces the previously rejected attempt rather than creating a duplicate - note that a rejected record was never stored, so on re-submission it will be treated as a Create, not an Update.
• Partial-file failures will not halt processing of the remaining lines in that file.

4.1.3.5. Supported Profiles

4.1.3.6. Extensions

4.1.3.7. Expected Behavior

Per-record behavior within a bulk load batch is expected to mirror some of the items defined in the Submit Immunization interaction's Expected Behavior table (same OperationOutcome codes, business-rule codes, and HTTP status semantics per record), since each NDJSON line is processed as an individual conditional-update transaction entry.

In addition, the following bulk specific considerations apply: