Interoperability: Dataset-JSON and Pandas

odmlib v0.2.0 adds two new interoperability features:

Dataset-JSON v1.1 support — create, read, write, and convert CDISC Dataset-JSON documents
Optional Pandas DataFrame integration — export metadata and data to DataFrames, and import DataFrame rows back into odmlib objects

—

Working with Dataset-JSON

CDISC Dataset-JSON is a column-oriented JSON format for regulatory submission datasets. odmlib provides a pure-Python model that requires no additional dependencies.

Creating a Dataset-JSON document

from odmlib.dataset_json_1_1 import DatasetJSON, Column

# Create columns
columns = [
    Column(itemOID="IT.STUDYID", name="STUDYID", label="Study Identifier", dataType="string"),
    Column(itemOID="IT.USUBJID", name="USUBJID", label="Unique Subject Identifier", dataType="string"),
    Column(itemOID="IT.AETERM", name="AETERM", label="Reported Term for the AE", dataType="string"),
    Column(itemOID="IT.AESEQ", name="AESEQ", label="Sequence Number", dataType="integer"),
]

# Create the dataset
ae = DatasetJSON(
    datasetJSONCreationDateTime="2024-01-15T12:00:00",
    datasetJSONVersion="1.1.0",
    fileOID="ODM.DATASET.AE",
    studyOID="CDISC01",
    metaDataVersionOID="MDV.001",
    originator="ACME Corp",
    name="AE",
    label="Adverse Events",
    columns=columns,
    records=10,
    rows=[
        ["CDISC01", "CDISC01-001", "Headache", 1],
        ["CDISC01", "CDISC01-001", "Nausea", 2],
    ],
)
ae.write_json("ae_dataset.json")

Reading a Dataset-JSON file

from odmlib.dataset_json_1_1 import DatasetJSON

ae = DatasetJSON.read_json("ae_dataset.json")
print(f"File OID: {ae.fileOID}")
print(f"Columns: {ae.column_names}")
print(f"Rows: {len(ae.rows)}")

Converting from Dataset-XML

Use odmlib.dataset_json_1_1.converter.dataset_xml_to_dataset_json() to convert an odmlib Dataset-XML 1.0.1 object to Dataset-JSON v1.1:

import odmlib.odm_loader as OL
import odmlib.loader as LD
import odmlib.ns_registry as NS
from odmlib.dataset_json_1_1.converter import dataset_xml_to_dataset_json

# Set up namespaces for Dataset-XML
NS.NamespaceRegistry(prefix="odm",  uri="http://www.cdisc.org/ns/odm/v1.3",
                     is_default=True)
NS.NamespaceRegistry(prefix="data", uri="http://www.cdisc.org/ns/Dataset-XML/v1.0")

# Load the Dataset-XML file
loader = LD.ODMLoader(OL.XMLODMLoader(
    model_package="dataset_1_0_1",
    ns_uri="http://www.cdisc.org/ns/Dataset-XML/v1.0",
))
loader.open_odm_document("ae.xml")
odm = loader.root()

# Convert to Dataset-JSON v1.1 (returns dict of dataset name -> DatasetJSON)
datasets = dataset_xml_to_dataset_json(odm)
for name, ds in datasets.items():
    ds.write_json(f"{name}_dataset.json")

Optionally pass a Define-XML MetaDataVersion to enrich column metadata (labels, types, lengths):

from odmlib.dataset_json_1_1.converter import dataset_xml_to_dataset_json

# Load Define-XML for metadata enrichment
define_loader = LD.ODMLoader(DL.XMLDefineLoader(model_package="define_2_1"))
define_loader.open_odm_document("define.xml")
mdv = define_loader.MetaDataVersion()

# Convert with enriched metadata
datasets = dataset_xml_to_dataset_json(odm, define_mdv=mdv)

—

Using Pandas with odmlib

Install Pandas (optional dependency):

pip install odmlib[dataframe]
# or
pip install pandas

Exporting metadata to a DataFrame

import odmlib.loader as LD
import odmlib.define_loader as DL
from odmlib.dataframe import metadata_to_dataframe

loader = LD.ODMLoader(DL.XMLDefineLoader(model_package="define_2_1"))
loader.open_odm_document("define.xml")
mdv = loader.MetaDataVersion()

# All ItemDef attributes as a DataFrame
df = metadata_to_dataframe(mdv, "ItemDef")
print(df[["OID", "Name", "DataType", "Length"]].head(10))

# Filter to specific columns only
df_slim = metadata_to_dataframe(mdv, "ItemDef", attributes=["OID", "Name", "DataType"])

Exporting clinical data to a DataFrame

For ODM 1.3.2 hierarchical clinical data:

from odmlib.dataframe import clinical_data_to_dataframe

# Flatten all VS records to a DataFrame
df = clinical_data_to_dataframe(odm.ClinicalData, item_group_oid="IG.VS")
print(df.head())

For Dataset-XML 1.0.1 (flat structure):

from odmlib.dataframe import dataset_to_dataframe

df = dataset_to_dataframe(odm.ClinicalData)
print(df.head())

For Dataset-JSON v1.1 datasets:

from odmlib.dataset_json_1_1 import DatasetJSON
from odmlib.dataframe import dataset_json_to_dataframe

ae = DatasetJSON.read_json("ae_dataset.json")
df = dataset_json_to_dataframe(ae)
print(df.describe())

Importing from a DataFrame back into odmlib

import odmlib.define_2_1.model as DEF
import pandas as pd
from odmlib.dataframe import dataframe_to_items

# Load item definitions from a spreadsheet or CSV
df = pd.read_excel("items.xlsx")

# Map spreadsheet columns to odmlib attribute names if needed
mapping = {"Variable Name": "Name", "Item OID": "OID", "Type": "DataType"}

items = dataframe_to_items(df, DEF, "ItemDef", column_mapping=mapping)
for item in items:
    mdv.ItemDef.append(item)

Note

dataframe_to_items() skips rows that cannot be constructed as valid odmlib elements (e.g., rows with missing required attributes). Check the returned list length against the input DataFrame length to detect skipped rows.