Omni Data Refinement Examples

The following examples demonstrate how to use OMR for various data quality, analysis, and validation workflows. You can find these original python files in the examples/ directory of the repository.

01. Dataset Summary

"""
OMR Example 01: Dataset Summary
================================
The first step when working with any new dataset is getting a quick
overview of its structure and quality. OMR's `.summary()` method provides
a single-line snapshot covering: shape, missing value percentage, duplicate
row count, and an overall heuristic health score.

This replaces the need to chain pandas `.info()`, `.describe()`, and
manual null-checking calls.
"""

import pandas as pd
from omr import Dataset

# A realistic dataset containing missing values and an outlier in 'age'.
df = pd.DataFrame({
    "age": [25, 30, None, 40, 40, 150],  # None is missing; 150 is a likely outlier
    "income": [50000, 60000, 70000, None, None, 90000],  # Two missing values
    "city": ["NY", "LA", "NY", "SF", "SF", None]  # One missing value
})

# Wrap the raw DataFrame in an OMR Dataset.
# This enables all OMR quality, profiling, and validation features.
dataset = Dataset(df)

# Call .summary() to print a compact, single-line overview of the dataset.
# The health score ranges from 0 (poor) to 100 (excellent).
# A green score means the data is safe to use.
# A red score means issues were detected — use .health() to investigate further.
print("Executing dataset.summary()...\n")
dataset.summary()

02. Health Check

"""
OMR Example 02: Health Check
==============================
When the summary health score is low, use `.health()` to get a detailed
breakdown of exactly what is wrong with the data.

.health() evaluates data across five quality pillars:
  - Completeness  : Are there missing values?
  - Uniqueness    : Are there duplicate rows or IDs?
  - Consistency   : Are data types uniform within each column?
  - Validity      : Are values within acceptable ranges?
  - Conformity    : Do values conform to expected patterns?

The method returns a HealthReport object that can be used programmatically
to enforce quality gates in CI/CD pipelines.
"""

import pandas as pd
from omr import Dataset

# A dataset with multiple data quality issues:
#   - customer_id 3 appears twice (duplicate)
#   - age -5 is logically invalid; 120 is an extreme outlier; None is missing
#   - salary column mixes integers and strings ("70k"), causing type inconsistency
df = pd.DataFrame({
    "customer_id": [1, 2, 3, 3, 5],
    "age": [25, 30, -5, 120, None],
    "salary": [50000, 60000, "70k", 80000, 90000],
    "is_active": [1, 0, 1, 0, 1]
})

dataset = Dataset(df)

# Run the full health check across all five quality pillars.
# The output includes a score out of 100, a per-pillar breakdown,
# and a list of specific issues with severity levels and recommended fixes.
print("Executing dataset.health()...\n")
report = dataset.health()

# The report object can be accessed programmatically.
# Use the score to enforce a quality gate: if the score is below 80,
# flag the dataset as unfit and stop the pipeline.
print(f"\n[Programmatic Access] Raw score: {report.score}")
if report.score < 80:
    print("[Pipeline Alert] Health score is too low. Data requires cleaning before use.")

03. Column Profiling

"""
OMR Example 03: Column Profiling
==================================
.profile() generates a unified statistical profile for every column in the
dataset, regardless of data type.

For numeric columns it reports: mean, min, max, standard deviation,
missing count, and unique value count.

For categorical columns it reports: top/most frequent value, unique count,
and missing count.

This is a single-call replacement for pandas .info() + .describe(), which
ignores categorical columns entirely.
"""

import pandas as pd
from omr import Dataset

# A mixed dataset with numeric, categorical, and boolean columns.
# The 'price' column has one missing value to demonstrate null reporting.
df = pd.DataFrame({
    "product_id": [101, 102, 103, 104, 105],
    "category": ["Electronics", "Clothing", "Electronics", "Home", "Clothing"],
    "price": [299.99, 45.50, 899.00, 120.00, None],  # One missing value
    "in_stock": [True, True, False, True, False]
})

dataset = Dataset(df)

# Generate and print the full column profile.
# Each column is displayed as a formatted row in a terminal table showing
# its type, statistical metrics, missing count, and unique value count.
print("Executing dataset.profile()...\n")
dataset.profile()

04. Automatic Cleaning

"""
OMR Example 04: Automatic Cleaning
=====================================
.clean() runs OMR's intelligent cleaning engine, which reads the internal
health report and applies a set of automatic fixes:

  - Missing numeric values  : imputed using the column median
  - Missing categorical values : filled with a constant placeholder
  - Duplicate rows          : dropped, keeping the first occurrence
  - Mixed-type columns      : cast to a safe, uniform string type

Call .export() after .clean() to retrieve the cleaned data as a standard
pandas DataFrame.
"""

import pandas as pd
from omr import Dataset

# A dataset with all four common data quality issues:
#   - 'id' column contains a duplicated value (2 appears twice)
#   - 'department' has a missing categorical value
#   - 'salary' has a missing numeric value
#   - 'bonus' mixes integers and a string ("500"), causing a type conflict
df = pd.DataFrame({
    "id": [1, 2, 2, 4, 5],
    "department": ["HR", "IT", "IT", None, "Sales"],
    "salary": [50000, 60000, 60000, None, 90000],
    "bonus": [1000, "500", 500, 2000, 3000]
})

print("BEFORE CLEANING:")
print(df)
print("-" * 50)

dataset = Dataset(df)

# Run the automatic cleaning engine.
# The engine inspects each column's detected issues and applies the
# appropriate fix. No manual configuration is required.
print("\nExecuting dataset.clean()...\n")
dataset.clean()

# Export the cleaned data back to a pandas DataFrame for downstream use.
clean_df = dataset.export()

print("AFTER CLEANING:")
print(clean_df)
print("-" * 50)
# Expected results:
#   - The duplicate row (id=2) has been removed
#   - The missing salary has been filled with the column median
#   - The missing department has been filled with a constant
#   - The 'bonus' column is now uniformly typed as string

05. Explain Changes

"""
OMR Example 05: Explain Changes (Transformation Log)
=======================================================
Every operation performed by .clean() is strictly logged internally.
.explain_changes() prints this transformation log, providing full
auditability of what was changed, which columns were affected, how many
rows were touched, and the reasoning behind each decision.

This is essential in regulated or enterprise environments where data
transformations must be documented and explainable.
"""

import pandas as pd
from omr import Dataset

# A small dataset with a duplicate row and a missing numeric value.
df = pd.DataFrame({
    "id": [1, 2, 2, 4],
    "sales": [100, None, 100, 400]
})

dataset = Dataset(df)

# Run health detection first, then apply automatic cleaning.
# Both steps are required before calling explain_changes().
dataset.health()
dataset.clean()

# Print the transformation log.
# Each entry in the log describes:
#   - Column name
#   - Action taken (e.g., Median Imputation, Duplicate Removal)
#   - Number of rows affected
#   - The reason the action was triggered
print("\nExecuting dataset.explain_changes()...\n")
dataset.explain_changes()

06. Statistical Analysis

"""
OMR Example 06: Statistical Analysis
=======================================
A dataset can be structurally clean (no nulls, no duplicates, correct types)
but still be statistically problematic for machine learning models.

.analyze() runs OMR's statistics engine to detect deeper issues:
  - Outliers       : detected using Z-score and IQR methods
  - Skewness       : identifies heavily skewed distributions that can
                     bias linear and tree-based models
  - Multicollinearity : detects highly correlated feature pairs
  - Class imbalance   : flags target columns with unequal class distribution

Each detected issue includes a severity level and a recommended action.
"""

import pandas as pd
from omr import Dataset

# A dataset with no missing values but with statistical problems:
#   - 'age' contains 150, a clear extreme outlier
#   - 'income' is heavily right-skewed due to the 900000 value
df = pd.DataFrame({
    "age":    [22, 25, 23, 24, 22, 21, 26, 25, 24, 150],
    "income": [40000, 42000, 45000, 41000, 43000, 40000, 44000, 46000, 42000, 900000],
})

dataset = Dataset(df)

# Run the statistical analysis engine.
# The output highlights flagged columns with their detected issue,
# severity, and a concrete recommendation (e.g., "Apply log transform").
print("Executing dataset.analyze()...\n")
dataset.analyze()

07. Compare Datasets

"""
OMR Example 07: Dataset Comparison and Drift Detection
=========================================================
Data drift occurs when the statistical distribution of production data
shifts away from the distribution the model was trained on. This causes
model predictions to silently degrade over time without any visible error.

.compare(other_dataset) detects drift between two datasets by running:
  - Population Stability Index (PSI)
  - Kolmogorov-Smirnov test
  - Jensen-Shannon divergence

Each column receives a drift severity: None, Low, Medium, or High.
A High severity result means the column's distribution has fundamentally
changed and the model should be retrained.
"""

import pandas as pd
from omr import Dataset

# The original training dataset — this represents the data distribution
# the model learned from during training.
training_df = pd.DataFrame({
    "age":    [20, 22, 25, 23, 26],
    "salary": [40000, 42000, 45000, 41000, 46000]
})
training_dataset = Dataset(training_df)

# The incoming production dataset — represents data arriving today.
# 'age' has shifted slightly, but 'salary' has approximately doubled,
# which would cause significant model degradation.
production_df = pd.DataFrame({
    "age":    [21, 23, 26, 24, 28],
    "salary": [80000, 84000, 90000, 82000, 92000]
})
production_dataset = Dataset(production_df)

# Compare the training dataset against the production dataset.
# The drift report shows each column's PSI score and severity.
# If 'salary' is flagged as High, the model must be retrained.
print("Executing training_dataset.compare(production_dataset)...\n")
training_dataset.compare(production_dataset)

08. Explain Concepts

"""
OMR Example 08: Built-in Concept Explainer
============================================
When OMR detects an issue such as data leakage or class imbalance, it
surfaces a keyword describing the problem. .explain(keyword) looks up that
keyword and returns a structured explanation covering:

  - Definition    : What the concept means
  - Why it matters: How it affects model training and performance
  - Risks         : Specific failure modes it causes
  - Fixes         : A numbered list of recommended remediation strategies

Supported keywords include: 'class_imbalance', 'data_leakage',
'multicollinearity', 'high_cardinality', 'skewness', and more.
"""

import pandas as pd
from omr import Dataset

# A minimal dataset is needed to initialize the Dataset object.
# The .explain() method does not use the data — it only needs the keyword.
dataset = Dataset(pd.DataFrame({"a": [1]}))

# Look up an explanation for 'class_imbalance'.
# Returns a formatted panel with definition, risks, and recommended fixes.
print("Executing dataset.explain('class_imbalance')...\n")
explanation = dataset.explain("class_imbalance")

# Look up an explanation for 'data_leakage'.
# Data leakage is one of the most common causes of overfitting in ML pipelines.
print("\nExecuting dataset.explain('data_leakage')...\n")
dataset.explain("data_leakage")

09. Schema Validation

"""
OMR Example 09: Schema Validation
====================================
.validate(rules) enforces business logic rules against the dataset.
It checks that column values satisfy the constraints you define, not just
that they are structurally valid.

Rules are defined as a dictionary mapping column names to schema validators
from the `omr.schemas` module:

  - schemas.PositiveInteger(min, max) : enforces a numeric range
  - schemas.Email()                   : validates email format via regex
  - schemas.OneOf(*values)            : restricts to an allowed set of values

.validate() reports exactly which rows failed each rule, the column involved,
and the invalid value found. This is used for enforcing data contracts at
pipeline entry points.
"""

import pandas as pd
from omr import Dataset, schemas

# A user dataset with several business rule violations:
#   - Row 1: age=17, below the minimum allowed age of 18
#   - Row 1: email='b.com', missing the '@' character (invalid format)
#   - Row 3: status='banned', not in the allowed values ('active', 'pending')
df = pd.DataFrame({
    "user_id": [1, 2, 3, 4],
    "age":     [25, 17, 30, 45],
    "email":   ["a@a.com", "b.com", "c@c.com", "d@d.com"],
    "status":  ["active", "active", "pending", "banned"]
})

dataset = Dataset(df)

# Define the business rules as a schema dictionary.
# Each key is a column name; each value is a validator instance.
business_rules = {
    "age":    schemas.PositiveInteger(min=18, max=120),
    "email":  schemas.Email(),
    "status": schemas.OneOf("active", "pending")
}

# Run validation against all defined rules.
# The output lists each failing row with the column name, the rule that
# was violated, and the actual value that caused the failure.
print("Executing dataset.validate()...\n")
dataset.validate(business_rules)

10. Versioning

"""
OMR Example 10: Versioning and Rollback
=========================================
OMR includes a built-in versioning system that saves in-memory snapshots
of the dataset state. This allows you to experiment freely with cleaning
or transformation operations and undo them instantly without reloading data
from disk.

Workflow:
  1. Call .snapshot() before a risky operation to save the current state.
     It returns a version_id string that references this snapshot.
  2. Apply cleaning or transformation operations.
  3. If the result is unsatisfactory, call .rollback(version_id) to restore
     the dataset to the exact state it was in when the snapshot was taken.
"""

import pandas as pd
from omr import Dataset

# A dataset with one missing value that will be imputed during cleaning.
df = pd.DataFrame({"data": [1, 2, 3, None, 5]})

dataset = Dataset(df)

# Save a snapshot before applying any transformation.
# The 'name' and 'description' parameters are optional labels for the log.
# The returned version_id is required to reference this snapshot later.
print("Taking snapshot...")
version_id = dataset.snapshot(name="Pre-Cleaning", description="Before imputation")
print(f"Saved Version ID: {version_id}")

print("\nBefore Cleaning:")
print(dataset.export())

# Apply the cleaning engine. This will impute the missing value.
dataset.clean()
print("\nAfter Cleaning:")
print(dataset.export())

# Restore the dataset to the pre-cleaning state using the saved version_id.
# This does not reload from disk — it reads from the in-memory snapshot.
print("\nExecuting dataset.rollback()...\n")
dataset.rollback(version_id)

print("After Rollback:")
print(dataset.export())
# The missing value (None) should be restored, confirming the rollback succeeded.

11. Report and Export

"""
OMR Example 11: Report Generation and Data Export
====================================================
After completing all quality checks and cleaning operations, OMR provides
two output mechanisms:

  .report(format, path)
      Generates a data quality report saved to disk. Supported formats:
        - "html"     : An interactive dashboard viewable in any browser.
        - "markdown" : A formatted report suitable for GitHub wikis or READMEs.
        - "json"     : A machine-readable report for automated pipelines.

  .export()
      Returns the current state of the dataset as a standard pandas DataFrame.
      The returned object is fully compatible with scikit-learn, PyTorch,
      and any other library that accepts a DataFrame or numpy array.
"""

import pandas as pd
from omr import Dataset

df = pd.DataFrame({
    "a": [1, 2, 3, 4],
    "b": ["x", "y", "x", "z"]
})

dataset = Dataset(df)

# Run a health check to populate the internal report data.
# .report() requires at least one analysis method to have been called first.
dataset.health()

# Generate reports in all three formats.
# Each call writes the report to disk and returns the file path.
print("\nGenerating Reports...")
html_path = dataset.report(format="html", path="my_report")
md_path   = dataset.report(format="markdown", path="my_report")
json_path = dataset.report(format="json", path="my_report")

print(f"HTML report saved to  : {html_path}")
print(f"Markdown report saved : {md_path}")
print(f"JSON report saved     : {json_path}")

# Export the dataset as a pandas DataFrame for use in downstream tasks.
# This creates no copy overhead — it returns the underlying DataFrame directly.
print("\nExecuting dataset.export()...\n")
clean_dataframe = dataset.export()

print("Exported type :", type(clean_dataframe))
print(clean_dataframe)

12. Monitoring

"""
OMR Example 12: Continuous Production Monitoring
==================================================
In production, data arrives continuously in batches (daily, hourly, etc.).
If an upstream pipeline breaks — for example, a column becomes entirely null
or a distribution shifts dramatically — the model will silently produce
incorrect predictions until the issue is manually discovered.

The OMR Monitor class enables automated quality surveillance:

  monitor.watch(baseline_dataset)
      Registers a baseline dataset. All future batches are compared
      against this reference distribution.

  monitor.check(new_batch_df)
      Compares an incoming DataFrame against the baseline. Returns a list
      of Alert objects, each containing:
        - severity  : the alert level (e.g., "HIGH", "MEDIUM")
        - check     : the name of the check that triggered the alert
        - message   : a human-readable description of the anomaly detected

Integrate monitor.check() into an Airflow DAG, a cron job, or any
data pipeline orchestrator to catch regressions before they reach the model.
"""

import pandas as pd
from omr import Dataset, Monitor

# Establish the baseline: the known-good distribution of the data.
# This is typically the dataset used to train or validate the model.
baseline_df = pd.DataFrame({"sales": [100, 110, 105, 95, 100]})
baseline_dataset = Dataset(baseline_df)

print("Initializing Monitor...")
monitor = Monitor()
monitor.watch(baseline_dataset)

# Simulate a broken upstream pipeline where sales values collapse to near zero.
# In a real system, this new_batch would come from a database query or file read.
print("\nNew daily batch arrives (simulating a broken pipeline)...")
new_batch = pd.DataFrame({"sales": [1, 0, 2, 0, 1]})

# Run the monitoring check.
# The monitor compares the new batch's statistics against the baseline.
# It detects the severe mean shift (from ~100 to ~0.8) and returns alerts.
print("Executing monitor.check(new_batch)...\n")
alerts = monitor.check(new_batch)

# Print each alert returned by the monitor.
# In production, pipe these alerts to Slack, PagerDuty, or an alerting system
# to notify engineers before the bad data propagates to downstream models.
for alert in alerts:
    print(f"[{alert.severity} ALERT] {alert.check}: {alert.message}")