Linkage with other systems (HMIS, CRVS, etc.)
The process of sharing data to external systems can be automated, manual, or a combination of manual and automated. In its most basic sense, data is exported from one system and imported into another. For SMSS systems, data is generally exported to an external system. Usually, this is a one-way synchronization, but there can be cases for importing data from external systems.
Any automated steps are generally handled using an Application Programming Interface (API). The API can be on the SMSS side , on the external system, or both.
Clarify Context and Substance of SMSS data
Before data can be linked, it’s important that external data managers, programmers, and users have a clear understanding of the background, context, and substance of the SMSS data. SMSS data is sampled data and is representative of the sampling frame. It is not comprehensive. For example, if the SMSS data shows that 100 deaths occurred in a province during 2023. It would be important to understand that those 100 deaths represent only a small percentage of total deaths within the province.
Additionally, the managers of the External System should provide context and substance of their system to the SMSS team. This will help guide the SMSS team in producing and shaping the data most efficiently and effectively for the External System.
Define Key elements of data sharing
- Structure of data to be shared
- Cycle of data sharing
- Incremental or Comprehensive
- Documentation
Structure of data to be shared
Both teams should work together to decide on appropriate data to be shared. Initially, the discussions can be high level, but the goal is to work towards an exact definition with all data points clearly defined. This is often referred to as a data model. As the teams work towards a defined data model, it is important to keep in mind the final audience for the external system. A good question might be: What data will be needed and how should it be shaped in order that the external users fully understand and make use of the data and information they receive?
A data model describes a group of data tables with clear definitions of every field (Name, data type, data values). Table keys and relationships should also be defined. Sample data should be provided early in the process to confirm understanding and to facilitate testing.
Data structure should also describe data format. Generally data will be shared using generic (CSV) or machine interpretable formats (JSON, XML).
An example of data structure shared from SMSS to an HMIS system is linked here (TODO: ADD LINK TO EXAMPLE DATA STRUCTURE).
Cycle of Data Sharing
This refers to the time frame or interval for sharing data (yearly, monthly, daily, real-time, etc). For a HMIS linkage, this might be an annual or quarterly synchronization. Real-time might be useful for CRVS data where a single new birth notification might be sent to the CRVS system whenever a new record arrives in the SMSS data system. Monthly might be useful in sharing lists of births and deaths to CRVS system to validate CRVS activity.
There are many factors that go into the decision on an appropriate cycle:
Data collection lag: Some data can take weeks or months to complete. For example, after an initial death notification, it can take weeks or months to complete a follow up verbal autopsy.
Burden on resources: Does generation of the dataset require data manager effort or CPU processing power?
Validity of time period: Are users expecting updates every month or year? For example, is there an established practice of reviewing data on quarterly basis or is data being prepared for an annual report to share with other ministries.
Based on those factors, the teams involved should consider what frequency is most appropriate for their cycle of data sharing.
Incremental or Comprehensive
Incremental: Sharing a specific timespan of data. For example, during the first year, you might share 2023 data. The next year you would share 2024 data, etc. In this case new data would be appended in the external system database.
Comprehensive: Sharing all relevant data. For example, during the first year, you would share 2023 data. The next year you would share 2023 and 2024 data. The next year 2023-2025 data. In this case, the data would overwrite previous data in the external system database.
Both options have merit. Incremental data can be more appropriate as data size grows larger over time. Comprehensive data can be more efficient in terms of simplicity and prevention of data falling into gaps.
Documentation
Documentation should include context and substance of the SMSS data for any users who are not familiar with the dataset. It should also include information about the data structure, sharing cycle, and protocol.
Define data sharing protocol
As the project grows, the data sharing protocol will grow and become more sophisticated. During development, this process will be mostly manual. Data managers and programmers will create data extracts. Fields will be added and formatting changed as needs are better understood and become more defined. Out of this intial cycle of planning, testing, and refactoring. The protocol will become more clear. At this point, the documentation should be updated. Elements include all the items above with a logical map describing: Context and Substrance, Data Structure and Time Cycle, and the newly developed Programming workflow. If time allows, detailed specifications can be documented.
Over time, the system will grow and the protocol documentation should grow with it.
Important Elements outside the scope of this document
Presenting SMSS data on the External System is generally the duty of the external team, but they may lack expertise in understanding SMSS data. It can be important for SMSS scientists and data managers to review and participate in designing the Graphs, Figures, Tables, and documentation that are used on the external system.
Data privacy and de-identification of data is an important consideration whenever sensitive data is shared. These issues are not described in this document.
| Last updated |
|---|
| May 21, 2025 |