Testing document fidelity: build an automated test suite for Office -> LibreOffice conversions

Why document-fidelity testing matters when moving from Microsoft 365 to LibreOffice (2026)

Legacy Office documents are everywhere in enterprise estates. When IT teams or governments choose LibreOffice to reduce licensing cost, increase privacy, or avoid vendor lock-in, the real work begins: ensuring documents render, behave, and automate the same way. In 2026, with LibreOffice and Microsoft both improving import/export filters and with organizations adopting hybrid cloud CI pipelines, you need an automated, repeatable approach to measure conversion fidelity — not manual spot checks.

What this guide covers

• How to design targeted test cases that cover formatting, layout, embedded objects, and macro behavior.
• How to build an automation harness using PowerShell, LibreOffice headless conversions, and CI pipelines.
• Which metrics actually measure fidelity (visual, structural, and functional).
• Practical scripts, comparisons and CI patterns you can adopt immediately.

Practical takeaway: Treat conversion testing like software QA — create a corpus, define acceptance criteria, automate conversions, and measure both pixel-level and semantic differences.

Trends shaping conversions in 2026

Late 2024 through 2026 saw meaningful improvements to Office interoperability. LibreOffice's import filters have continued to close the gap on OOXML edge cases and Office's support for ODF has been tightened in recent updates. At the same time, AI-powered content (smart art, generated charts) is more common inside documents, making fidelity testing both more crucial and more complex.

Operational teams running migrations now expect: automated regression suites integrated into CI, containerized LibreOffice instances for consistency, and quantifiable KPIs (not subjective "looks fine").

Designing test cases: cover the attack surface

Start by defining the document surface you care about. Break test cases into logical groups so you can prioritize. Below is a pragmatic taxonomy tailored for Office → LibreOffice migrations.

1. Formatting & layout

• Basic text: fonts, sizes, paragraph spacing, indents, line-height.
• Complex styles: nested styles, named styles, theme colors.
• Lists: multi-level bullets/numbering, custom list styles.
• Tables: merged cells, nested tables, table styles and borders.
• Page layout: margins, headers/footers, page breaks, sections, column layouts.

2. Graphics & embedded objects

• Images: inline vs anchored, transparency, DPI and scaling.
• Shapes/Vectors: SmartArt, Office shapes, OLE objects.
• Charts: embedded Excel charts vs exported images.

3. Data & spreadsheets

• Formulas: compatibility of functions and reference evaluation.
• Named ranges, pivot tables, data connections.
• Formatting: conditional formatting and custom number formats.

4. Macros & automation

• VBA macros in Word/Excel/PowerPoint and how LibreOffice Basic or script bridges handle them.
• Macro security prompts and signed macro behavior.
• Macro conversion attempts, execution success and behavioral equivalence.

5. Metadata, accessibility & language

• Document properties, revision history, tracked changes, comments.
• Right-to-left text, language tags, hyphenation, Unicode normalization.
• Accessibility tags and alt text preservation.

Constructing a representative test corpus

Your corpus should include real-world samples and synthetic edge cases. Aim for a blend of:

• Top 50 real documents by business importance (policy docs, contracts, templates).
• Category-specific test documents: very long docs, large tables, complex spreadsheets, slide decks with animations.
• Fuzzed documents: programmatically generated documents to hit corner cases (deep nested lists, extreme style names).

Store test documents in version control, tag them by category and expected behavior, and add metadata for priority and risk.

Automation harness architecture — practical blueprint

Below is a minimal viable automation harness that can scale into CI. The goal is repeatable conversions, standardized extraction, and automated comparisons that produce quantitative metrics.

Core components

• Conversion runner: uses LibreOffice headless (soffice) inside containers to convert docx → odt and docx → pdf.
• Extraction & normalization: unzip ODT/DOCX, normalize content.xml and document.xml (namespace prefixes, timestamp removal).
• Comparators: semantic (XML path diffs, style name mapping), textual (word diffs), visual (PDF → PNG raster diff with SSIM).
• Macro test harness: sandboxed Windows runner or Wine container to execute macros and capture outputs/side-effects.
• Metrics collector: aggregates scores per test and produces dashboards and pass/fail status for CI.

High-level flow

1. Checkout test corpus from Git.
2. Run conversion: original Office -> LibreOffice (docx -> odt & pdf).
3. Extract and normalize artifacts.
4. Run three comparators: structure, text, visual.
5. Execute macros where applicable and record success/failure and outputs.
6. Compute composite fidelity scores and store results.

PowerShell examples: headless conversion and extraction

PowerShell is the glue for Windows-based CI and admin teams. Below are practical snippets you can drop into a pipeline.

Converting a DOCX to ODT and PDF using soffice

$docx = 'C:\tests\sample.docx'
$workdir = 'C:\tests\out'
New-Item -ItemType Directory -Force -Path $workdir | Out-Null
$soffice = 'C:\Program Files\LibreOffice\program\soffice.exe'
& "$soffice" --headless --convert-to odt --outdir $workdir $docx
& "$soffice" --headless --convert-to pdf --outdir $workdir $docx

If you run LibreOffice in Docker/Linux CI, replace the path and call soffice inside the container.

Extract and normalize content.xml from an ODT

$odt = 'C:\tests\out\sample.odt'
$extract = 'C:\tests\out\extracted'
New-Item -ItemType Directory -Force -Path $extract | Out-Null
Add-Type -AssemblyName System.IO.Compression.FileSystem
[System.IO.Compression.ZipFile]::ExtractToDirectory($odt, $extract)
# Load content.xml, remove timestamps and normalize namespaces
[xml]$xml = Get-Content -Path (Join-Path $extract 'content.xml')
# Example normalization: remove meta:initial-creator and date attributes
$xml.SelectNodes('//*[@meta:initial-creator or @dc:date]', $null) | ForEach-Object { $_.RemoveAttribute('meta:initial-creator'); $_.RemoveAttribute('dc:date') }
$xml.Save((Join-Path $extract 'content.normalized.xml'))

Comparators and metrics — what to measure

Do not rely on a single metric. Combine visual, textual and structural measures into a composite fidelity score.

Visual comparison

Convert both source and converted artifacts to PDF (or rasterize pages) and compute pixel-level similarity. Use ImageMagick or pdiff implementations that compute SSIM (structural similarity) and per-page delta. Key outputs:

• SSIM per page (0.0–1.0, 1.0 is identical).
• Pixel-diff percentage per page.
• Number of pages with differences exceeding a threshold.

Textual comparison

Extract plain text from both versions (pdftotext, antiword, or XML extraction) and run word-level diffs. Useful metrics:

• Word match percentage.
• Normalized Levenshtein distance.
• Missing sections or truncated content flags.

Structural & style comparison

Compare the document XML elements and style definitions. This is where you catch missing style mappings, changed list levels, or table cell attribute differences. Metrics:

• Style mismatch ratio: count of styles with changed names or properties.
• Element path differences: number of text nodes reparented or relocated.
• Table cell format delta: border/merge/width mismatches.

Macro fidelity

Macro testing is the hardest but most consequential. You should measure both conversion attempts (was the macro converted/transcribed) and execution equivalence (did it behave correctly).

• Conversion presence: macro modules detected in the converted file.
• Execution success: macro ran without runtime errors under sandboxed execution.
• Behavioral equivalence: validation of macro side-effects (created docs, cell values, generated charts).

Example approach: run macros inside an isolated Windows VM or a Wine/mono environment, capture logs, and compare outputs to a golden run from Microsoft Office.

Composite fidelity score — turning metrics into decisions

Create a weighted scoring model to produce a single composite fidelity score per document. Example weights (customize to your priorities):

• Visual similarity: 40%
• Text match: 20%
• Style/structure parity: 20%
• Macro fidelity: 20% (or higher for macro-heavy estates)

Define acceptance thresholds: e.g., composite score >= 0.90 = Accept, 0.80–0.90 = Manual review, <0.80 = Fail/Remediate.

Macro test harness: practical considerations

Macro support is the reason many enterprises delay migration. LibreOffice can import VBA to LibreOffice Basic sometimes, but equivalence is not guaranteed. A test harness should:

• Run macros in a sandboxed Windows VM or container with a licensed Office install to capture baseline behavior.
• Run macros in LibreOffice (or a converted script) and compare side-effects.
• Capture return values, generated files, and any GUI prompts.
• Record macro security warnings; automated remediation may be to sign macros or adapt policies.

Automating macro execution and capture

Use PowerShell + COM automation on Windows to run a VBA macro and capture outputs:

# Run a macro in Word via COM and capture errors
$word = New-Object -ComObject Word.Application
$word.Visible = $false
$doc = $word.Documents.Open('C:\tests\sample.docm')
try {
  $word.Run('Module1.MacroMain')
  $result = 'Success'
} catch {
  $result = $_.Exception.Message
}
$doc.Close($false)
$word.Quit()
Write-Output $result

For LibreOffice, use the UNO bridge or run converted macro code. Log and compare the produced output files or values.

CI integration patterns

Integrate your harness into GitHub Actions, Azure DevOps, or GitLab CI. Recommended pattern:

1. On push to the corpus repo, trigger the conversion job in a Linux runner with a LibreOffice container.
2. Run comparators and upload artifacts (PDFs, normalized XML, diff images) to the pipeline artifacts store.
3. Publish a report with the composite score and failing tests as annotations.
4. Gate merges or deployments with the acceptance thresholds; require manual review for borderline cases.

Sample GitHub Actions snippet (conceptual)

name: Conversion Tests
on: [push]
jobs:
  convert-and-compare:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - name: Run LibreOffice conversion
        run: |
          docker run --rm -v ${{ github.workspace }}:/workspace libreoffice-image \
            soffice --headless --convert-to odt --outdir /workspace/out /workspace/corpus/*.docx
      - name: Run comparators
        run: |
          pwsh ./scripts/run-comparisons.ps1
      - name: Upload results
        uses: actions/upload-artifact@v4
        with: name: conversion-results
              path: out/reports

Reporting and dashboards

Store test results in a time-series store (Elastic, Prometheus + Grafana, or even a SQL DB) and track:

• Pass/fail rates over time and by document category.
• Common failure types (fonts, lists, macros).
• Regression alerts when a new LibreOffice release introduces breakage.

Operational tips and best practices

• Version pinning: Run conversions against specific LibreOffice builds in CI to avoid drift. Use container images with explicit tags.
• Golden baselines: Keep golden PDFs and XML outputs for comparison. When acceptable differences change, update goldens via a controlled PR with reviewers.
• Incremental migrations: Start with low-risk documents and iterate on templates that fail tests.
• Macro remediation plan: For critical VBA, plan to rewrite into scripts using Python + UNO, or keep a small subset of Office on locked-down VM hosts.
• Security: Treat macros as code. Enforce signing and scanning before allowing execution in automated harnesses.

Case study (mini)

In late 2025 a public-sector organization ran a pilot to move 10,000 policy docs to LibreOffice. They built a harness similar to the one described here and discovered:

• 60% of documents passed automated visual and structural checks.
• 25% required template cleanup (styles and fonts).
• 15% were macro-heavy and required either a macro rewrite or retention on Office-compatible readers.

By prioritizing high-value templates for remediation and automating the rest, they reduced projected licensing spend by 40% while retaining operational continuity.

Future predictions & where to invest in 2026

Expect continued improvements to OOXML filters and better tooling around macro translation in 2026 and beyond. Invest in:

• Automated visual diff tooling that supports SSIM and perceptual hashing.
• Macro migration toolchains (VBA → Python/UNO) aided by AI-assisted code suggestions, but validated in harnesses.
• Standardized corpora and open reports — the community will increasingly publish conversion test suites to help other organizations.

Wrap-up — checklist to get started

1. Assemble a representative test corpus and store it in Git.
2. Build conversion runner using headless LibreOffice in containers.
3. Implement three comparators: visual (SSIM), textual (word diff), structural (XML diff).
4. Create a macro sandbox to run and validate macro behavior.
5. Integrate into CI and set acceptance thresholds.

Final thoughts

Conversion is not a single event — it’s a product-quality activity. Automation, clear metrics and incremental remediation are the levers that make a migration to LibreOffice predictable and measurable. In 2026, with better filters and more tooling, the difference between a successful transition and a disruptive one will be how well you test and quantify fidelity.

Call to action

If you want a ready-made starting point, clone the reference harness we publish (includes PowerShell scripts, Dockerfiles, and example comparators) and run it against a small corpus this week. Share your toughest conversion case and I’ll outline a targeted remediation plan.

Related Reading

• Fonts for Sports Dashboards: What FPL Site Editors Need to Know
• Protecting Art and Heirlooms: UV-Blocking Curtains for Priceless Pieces
• Creating An Accessible Bartop Cabinet: Lessons from Sanibel’s Design Philosophy
• How to Stage an Easter Photoshoot Using RGB Lighting and Cozy Props
• Phone Plans, CRMs, and Budgeting: Building a Cost-Efficient Communications Stack for Your LLC