Tutorial 1: Merge Multi-ROI Chromatin Trace Data

This tutorial walks through the standard workflow for merging chromatin tracing data from multiple regions of interest (ROIs):

1. Collect trace files from all ROIs into one folder

2. Assess quality by computing Pearson correlations between ROIs

3. Remove ROIs with poor correlation (outliers)

4. Re-assess the correlation matrix after removal

5. Merge the remaining trace files into a single table

6. Statistics on the merged dataset

7. Next steps — link to Tutorial 2 for quality control

Step 0: Set-up your data and output path

import os
from pathlib import Path
import numpy as np
import matplotlib.pyplot as plt
import matplotlib.image as mpimg

Set up your folder data path:

data_path = "/mnt/grey/DATA/users/zidoum/traceraptops/jupyterlab_amel"

Set up destination folder for output:

dest_path = f"{data_path}/output"

Check ROIs detected:

print(f"Data path: {data_path}")
print(f"Output path: {dest_path}")
print(f"\nAvailable ROIs:")
for d in sorted(Path(data_path).iterdir()):
    if d.is_dir() and "ROI" in d.name:
        print(f"  {d.name}")

Data path: /mnt/grey/DATA/users/zidoum/traceraptops/jupyterlab_amel
Output path: /mnt/grey/DATA/users/zidoum/traceraptops/jupyterlab_amel/output

Available ROIs:
  ROI_51
  ROI_52
  ROI_53
  ROI_54

Step 1: Collect trace files from all ROIs

collect_files scans each subdirectory of --root for a file matching --example-file. The --variable-part "13" indicates that the ROI number varies; fixed-length matching naturally excludes Matrix files (different filename length).

!collect_files --root {data_path} --example-file "Trace_3D_barcode_mask-mask0_ROI-51.ecsv" --variable-part "51" --copy-to {dest_path}/raw_traces --force

------- Running collect_files.py --------
Collect one file per subdirectory using pattern matching.

Matched (4):
  ROI_51 -> /mnt/grey/DATA/users/zidoum/traceraptops/jupyterlab_amel/ROI_51/Trace_3D_barcode_mask-mask0_ROI-51.ecsv
  ROI_52 -> /mnt/grey/DATA/users/zidoum/traceraptops/jupyterlab_amel/ROI_52/Trace_3D_barcode_mask-mask0_ROI-52.ecsv
  ROI_53 -> /mnt/grey/DATA/users/zidoum/traceraptops/jupyterlab_amel/ROI_53/Trace_3D_barcode_mask-mask0_ROI-53.ecsv
  ROI_54 -> /mnt/grey/DATA/users/zidoum/traceraptops/jupyterlab_amel/ROI_54/Trace_3D_barcode_mask-mask0_ROI-54.ecsv
No match (1):
  output

Copied 4 file(s) to /mnt/grey/DATA/users/zidoum/traceraptops/jupyterlab_amel/output/raw_traces

Step 2: Compute Pearson correlations between ROIs

trace_pearsons computes a pairwise distance map for each ROI (median 3D distance between every barcode pair), then calculates the Pearson correlation between these maps.

A high correlation between two ROIs means they share similar chromatin organization. An ROI with low correlation against all others is likely an outlier (imaging artifact, poor segmentation, etc.).

!ls {dest_path}/raw_traces/*.ecsv | trace_pearsons --pipe -O {dest_path}

# Display the correlation matrix
img = mpimg.imread(f"{dest_path}/trace_correlation_matrix.png")
fig, ax = plt.subplots(figsize=(10, 10))
ax.imshow(img)
ax.axis('off')
plt.tight_layout()
plt.show()

------- Running trace_pearsons.py --------
Compare chromatin trace tables by computing pairwise distances.

Analyzing 4 trace files...
Processing Trace_3D_barcode_mask-mask0_ROI-51.ecsv
Processing Trace_3D_barcode_mask-mask0_ROI-52.ecsv
Processing Trace_3D_barcode_mask-mask0_ROI-53.ecsv
Processing Trace_3D_barcode_mask-mask0_ROI-54.ecsv
$ Saved correlation matrix as /mnt/grey/DATA/users/zidoum/traceraptops/jupyterlab_amel/output/trace_correlation_matrix.png
$ Saved correlation matrix data in NPY format: /mnt/grey/DATA/users/zidoum/traceraptops/jupyterlab_amel/output/trace_correlation_matrix.npy

Step 3: Remove poorly-correlated ROIs

For each ROI, we compute its mean Pearson correlation with all other ROIs. ROIs below the threshold are removed from the working folder before merging.

A threshold of 0.70 is conservative: it removes only clear outliers while preserving the vast majority of the data.

# Load the correlation matrix saved by trace_pearsons
corr = np.load(f"{dest_path}/trace_correlation_matrix.npy")
traces = sorted(Path(f"{dest_path}/raw_traces").glob("*.ecsv"))

# Mean correlation per ROI (exclude self-correlation on the diagonal)
np.fill_diagonal(corr, np.nan)
mean_corr = np.nanmean(corr, axis=1)

# Remove ROIs below threshold
threshold = 0.10
for trace, mc in zip(traces, mean_corr):
    if mc < threshold or np.isnan(mc):
        trace.unlink()
        print(f"Removed {trace.name}  (mean Pearson = {mc:.3f})")
    else:
        print(f"Kept    {trace.name}  (mean Pearson = {mc:.3f})")

Kept    Trace_3D_barcode_mask-mask0_ROI-51.ecsv  (mean Pearson = 0.721)
Kept    Trace_3D_barcode_mask-mask0_ROI-52.ecsv  (mean Pearson = 0.761)
Kept    Trace_3D_barcode_mask-mask0_ROI-53.ecsv  (mean Pearson = 0.760)
Kept    Trace_3D_barcode_mask-mask0_ROI-54.ecsv  (mean Pearson = 0.729)

Step 4: Re-run Pearson to verify improvement

After removing the outlier ROIs, the correlation matrix should show higher overall values.

!ls {dest_path}/raw_traces/*.ecsv | trace_pearsons --pipe -O {dest_path}

# Display the updated correlation matrix
img = mpimg.imread(f"{dest_path}/trace_correlation_matrix.png")
fig, ax = plt.subplots(figsize=(10, 10))
ax.imshow(img)
ax.axis('off')
plt.tight_layout()
plt.show()

------- Running trace_pearsons.py --------
Compare chromatin trace tables by computing pairwise distances.

Analyzing 4 trace files...
Processing Trace_3D_barcode_mask-mask0_ROI-51.ecsv
Processing Trace_3D_barcode_mask-mask0_ROI-52.ecsv
Processing Trace_3D_barcode_mask-mask0_ROI-53.ecsv
Processing Trace_3D_barcode_mask-mask0_ROI-54.ecsv
$ Saved correlation matrix as /mnt/grey/DATA/users/zidoum/traceraptops/jupyterlab_amel/output/trace_correlation_matrix.png
$ Saved correlation matrix data in NPY format: /mnt/grey/DATA/users/zidoum/traceraptops/jupyterlab_amel/output/trace_correlation_matrix.npy

Step 5: Merge trace files

!ls {dest_path}/raw_traces/*.ecsv | trace_merge -F {dest_path} -N merged_traces.ecsv

------- Running trace_merge.py --------
This script will merge trace tables provided as inputs.

Number of trace files to merge: 4
 $ Merged trace file will contain 83031 traces
Read and accumulated 4 trace files
$ Saving output table as /mnt/grey/DATA/users/zidoum/traceraptops/jupyterlab_amel/output/merged_traces.ecsv ...
Finished execution

Step 6: Compute basic statistics

!trace_stats --input {dest_path}/merged_traces.ecsv

------- Running trace_stats.py --------
Compute basic statistics for chromatin trace files.

$ Importing table from pyHiM format
Successfully loaded trace table: /mnt/grey/DATA/users/zidoum/traceraptops/jupyterlab_amel/output/merged_traces.ecsv
Statistics for /mnt/grey/DATA/users/zidoum/traceraptops/jupyterlab_amel/output/merged_traces.ecsv:
- Number of unique ROIs: 4
- Number of unique chromatin traces: 16777
- Number of unique barcodes: 31

Next steps

The merged trace file is ready for downstream analysis.

Continue with Tutorial 2 — Quality Control to:

• Generate detailed quality metrics with trace_analyzer

• Interpret barcode detection, neighbor distances, and barcode frequency plots

• Decide on filtering thresholds for Tutorial 3

Summary

Step	Script	Output
Collect	collect_files	raw_traces/ (one .ecsv per ROI)
Correlations	trace_pearsons	trace_correlation_matrix.png + .npy
Filter ROIs	Python (numpy)	removed outlier files from raw_traces/
Merge	trace_merge	merged_traces.ecsv
Statistics	trace_stats	stdout

Output location: data/output/