FedPyDESeq2 demo on the TCGA-LUAD dataset.

Note

Click here to download the full example code

This example demonstrates how to run a FedPyDESeq2 experiment on the TCGA-LUAD dataset from a single machine, using Substra's simulation mode.

We will show how to perform a simple differential expression analysis, comparing samples with "Advanced" vs "Non-advanced" tumoral stage.

from pathlib import Path

import pandas as pd
from fedpydeseq2_datasets.process_and_split_data import setup_tcga_dataset
from IPython.display import display

from fedpydeseq2.fedpydeseq2_pipeline import run_fedpydeseq2_experiment

Dataset setup

In a real federated setup, the data is distributed across multiple medical centers and must be registered with Substra beforehand. Hence, each center would have a folder containing two csvs (one fore the counts and one for the metadata), as well as an opener python file and a markdown readme file (see Substra's documentation on how to register a datasample). Then, we would only need pass the dataset_datasample_keys path.

In this tutorial, however, we use FedPyDESeq2's simulation mode, which allows us to emulate a federated setup from a single machine.

The simulation mode assumes the data to be organized in the following structure:

processed_data_path/
├── centers_data/
│   └── tcga/
│       └── {experiment_id}/
│           ├── center_0/
│           │   ├── counts.csv
│           │   └── metadata.csv
│           ├── center_1/
│           │   ├── counts.csv
│           │   └── metadata.csv
│           └── ...
└── pooled_data/
    └── tcga/
        └── {experiment_id}/
            ├── counts.csv
            └── metadata.csv

In this tutorial, we have already downloaded the data in the data/raw directory.

The setup_tcga_dataset function from fedpydeseq2_datasets will automatically organize the data in the data/processed directory.

It will split the TCGA-LUAD dataset into 7 centers according to the geographical origin of the samples, as described in the FedPyDESeq2 paper.

See also the fedpydeseq2_datasets repository for more details.

dataset_name = "TCGA-LUAD"
raw_data_path = Path("data/raw").resolve()
processed_data_path = Path("data/processed").resolve()
design_factors = "stage"

setup_tcga_dataset(
    raw_data_path,
    processed_data_path,
    dataset_name=dataset_name,
    small_samples=False,
    small_genes=False,
    only_two_centers=False,
    design_factors=design_factors,
    force=True,
)

experiment_id = "TCGA-LUAD-stage"

Out:

2025-05-13 08:02:32.910 | INFO     | fedpydeseq2_datasets.process_and_split_data:setup_tcga_dataset:144 - Setting up TCGA dataset: TCGA-LUAD-stage
2025-05-13 08:02:32.911 | INFO     | fedpydeseq2_datasets.process_and_split_data:setup_tcga_dataset:150 - First center metadata does not exist or force=True. Setting up the dataset.
2025-05-13 08:02:32.911 | INFO     | fedpydeseq2_datasets.process_and_split_data:_setup_tcga_dataset:282 - Processing the data for the TCGA dataset: TCGA-LUAD-stage
2025-05-13 08:02:35.527 | INFO     | fedpydeseq2_datasets.process_and_split_data:_setup_tcga_dataset:302 - Saving the data for each center /home/runner/work/fedpydeseq2/fedpydeseq2/docs/examples/data/processed/centers_data/tcga/TCGA-LUAD-stage
2025-05-13 08:03:16.099 | INFO     | fedpydeseq2_datasets.process_and_split_data:_setup_tcga_dataset:364 - Saving the pooled data at /home/runner/work/fedpydeseq2/fedpydeseq2/docs/examples/data/processed/pooled_data/tcga/TCGA-LUAD-stage

Running the experiment

We can now run the experiment.

Substra, the FL framework on which FedPyDESeq2 is built, supports a simulated mode which may be run locally from a single machine, which we will use here.

Let's run our FedPyDESeq2 experiment. This may be done using the run_fedpydeseq2_experiment wrapper function, which takes the following parameters:

n_centers=7: Our data is distributed across 7 different medical centers
backend="subprocess" and simulate=True: We'll run the analysis locally on our machine to simulate a federated setup, rather than in a real distributed environment
register_data=True: We'll register our dataset with Substra before analysis. In the case of a real federated setup, this would be set to False if data was already registered by Substra.
asset_directory: This directory should contain an opener.py file, containing an Opener class, and datasets.description.md file. Here, we copy them from fedpydeseq2_datasets/assets/tcga
centers_root_directory: Where the processed data for each center is stored
compute_plan_name: We'll call this analysis "Example-TCGA-LUAD-pipeline" in Substra
dataset_name: We're working with the TCGA-LUAD lung cancer dataset
design_factors: This should be a list of the design factors we wish to include in our analysis. Here, we're studying how "stage" (the cancer stage) affects gene expression
ref_levels: We're setting "Non-advanced" as our baseline cancer stage
contrast: This should be a list of three strings, of the form ["factor", "alternative_level", "baseline_level"]. To compare gene expression between "Advanced" vs "Non-advanced" stages, we set contrast=["stage", "Advanced", "Non-advanced"].
refit_cooks=True: After finding outliers using Cook's distance, we'll refit the model without them for more robust results

fl_results = run_fedpydeseq2_experiment(
    n_centers=7,
    backend="subprocess",
    simulate=True,
    register_data=True,
    asset_directory=Path("assets/tcga").resolve(),
    centers_root_directory=processed_data_path
    / "centers_data"
    / "tcga"
    / experiment_id,
    compute_plan_name="Example-TCGA-LUAD-pipeline",
    dataset_name="TCGA-LUAD",
    design_factors="stage",
    ref_levels={"stage": "Non-advanced"},
    contrast=["stage", "Advanced", "Non-advanced"],
    refit_cooks=True,
)

Out:

2025-05-13 08:03:56.521 | INFO     | fedpydeseq2.substra_utils.federated_experiment:run_federated_experiment:182 - Setting up organizations...
2025-05-13 08:03:56.523 | INFO     | fedpydeseq2.substra_utils.federated_experiment:run_federated_experiment:233 - Registering the datasets...
2025-05-13 08:03:56.523 | INFO     | fedpydeseq2.substra_utils.federated_experiment:run_federated_experiment:290 - Adding dataset to client MyOrg2MSP
2025-05-13 08:03:56.524 | INFO     | fedpydeseq2.substra_utils.federated_experiment:run_federated_experiment:295 - Dataset added. Key: fa75223b-5d4f-473e-ba52-e757e212e591
2025-05-13 08:03:56.524 | INFO     | fedpydeseq2.substra_utils.federated_experiment:run_federated_experiment:290 - Adding dataset to client MyOrg3MSP
2025-05-13 08:03:56.525 | INFO     | fedpydeseq2.substra_utils.federated_experiment:run_federated_experiment:295 - Dataset added. Key: b8e7c519-5239-4b37-8aed-ba6c70364d7e
2025-05-13 08:03:56.525 | INFO     | fedpydeseq2.substra_utils.federated_experiment:run_federated_experiment:290 - Adding dataset to client MyOrg4MSP
2025-05-13 08:03:56.525 | INFO     | fedpydeseq2.substra_utils.federated_experiment:run_federated_experiment:295 - Dataset added. Key: 5c0a1eb0-ecad-4104-840f-4c7c18a800d8
2025-05-13 08:03:56.526 | INFO     | fedpydeseq2.substra_utils.federated_experiment:run_federated_experiment:290 - Adding dataset to client MyOrg5MSP
2025-05-13 08:03:56.526 | INFO     | fedpydeseq2.substra_utils.federated_experiment:run_federated_experiment:295 - Dataset added. Key: 5ee72c8f-cdd6-43a6-8189-45763a5395d2
2025-05-13 08:03:56.527 | INFO     | fedpydeseq2.substra_utils.federated_experiment:run_federated_experiment:290 - Adding dataset to client MyOrg6MSP
2025-05-13 08:03:56.527 | INFO     | fedpydeseq2.substra_utils.federated_experiment:run_federated_experiment:295 - Dataset added. Key: af5816a7-53a1-474b-b88b-7d4c8735ce02
2025-05-13 08:03:56.527 | INFO     | fedpydeseq2.substra_utils.federated_experiment:run_federated_experiment:290 - Adding dataset to client MyOrg7MSP
2025-05-13 08:03:56.528 | INFO     | fedpydeseq2.substra_utils.federated_experiment:run_federated_experiment:295 - Dataset added. Key: 858976c0-b808-48b8-9da8-cbd4a968b717
2025-05-13 08:03:56.528 | INFO     | fedpydeseq2.substra_utils.federated_experiment:run_federated_experiment:290 - Adding dataset to client MyOrg8MSP
2025-05-13 08:03:56.529 | INFO     | fedpydeseq2.substra_utils.federated_experiment:run_federated_experiment:295 - Dataset added. Key: 145061ca-466b-42d0-ae36-3103d4b22ce5
2025-05-13 08:03:56.529 | INFO     | fedpydeseq2.substra_utils.federated_experiment:run_federated_experiment:316 - Datasets registered.
2025-05-13 08:03:56.529 | INFO     | fedpydeseq2.substra_utils.federated_experiment:run_federated_experiment:318 - Dataset keys: {'MyOrg2MSP': 'fa75223b-5d4f-473e-ba52-e757e212e591', 'MyOrg3MSP': 'b8e7c519-5239-4b37-8aed-ba6c70364d7e', 'MyOrg4MSP': '5c0a1eb0-ecad-4104-840f-4c7c18a800d8', 'MyOrg5MSP': '5ee72c8f-cdd6-43a6-8189-45763a5395d2', 'MyOrg6MSP': 'af5816a7-53a1-474b-b88b-7d4c8735ce02', 'MyOrg7MSP': '858976c0-b808-48b8-9da8-cbd4a968b717', 'MyOrg8MSP': '145061ca-466b-42d0-ae36-3103d4b22ce5'}
2025-05-13 08:04:20,969 - INFO - Simulating the execution of the compute plan.
2025-05-13 08:04:20.970 | INFO     | fedpydeseq2.core.deseq2_core.deseq2_full_pipe:run_deseq_pipe:66 - Building design matrices...
2025-05-13 08:04:21.406 | INFO     | fedpydeseq2.core.deseq2_core.deseq2_full_pipe:run_deseq_pipe:76 - Finished building design matrices.
2025-05-13 08:04:21.407 | INFO     | fedpydeseq2.core.deseq2_core.deseq2_full_pipe:run_deseq_pipe:82 - Computing size factors...
2025-05-13 08:04:21.859 | INFO     | fedpydeseq2.core.deseq2_core.deseq2_full_pipe:run_deseq_pipe:97 - Finished computing size factors.
2025-05-13 08:04:21.859 | INFO     | fedpydeseq2.core.deseq2_core.deseq2_full_pipe:run_deseq_pipe:101 - Running LFC and dispersions.
2025-05-13 08:04:21.859 | INFO     | fedpydeseq2.core.deseq2_core.deseq2_lfc_dispersions.deseq2_lfc_dispersions:run_deseq2_lfc_dispersions:94 - Fit genewise dispersions...
2025-05-13 08:09:31.113 | INFO     | fedpydeseq2.core.deseq2_core.deseq2_lfc_dispersions.deseq2_lfc_dispersions:run_deseq2_lfc_dispersions:108 - Finished fitting genewise dispersions.
2025-05-13 08:09:31.113 | INFO     | fedpydeseq2.core.deseq2_core.deseq2_lfc_dispersions.deseq2_lfc_dispersions:run_deseq2_lfc_dispersions:112 - Compute dispersion prior...
2025-05-13 08:09:34.429 | INFO     | fedpydeseq2.core.deseq2_core.deseq2_lfc_dispersions.deseq2_lfc_dispersions:run_deseq2_lfc_dispersions:125 - Finished computing dispersion prior.
2025-05-13 08:09:34.429 | INFO     | fedpydeseq2.core.deseq2_core.deseq2_lfc_dispersions.deseq2_lfc_dispersions:run_deseq2_lfc_dispersions:145 - Fit MAP dispersions...
2025-05-13 08:13:15.395 | INFO     | fedpydeseq2.core.deseq2_core.deseq2_lfc_dispersions.deseq2_lfc_dispersions:run_deseq2_lfc_dispersions:158 - Finished fitting MAP dispersions.
2025-05-13 08:13:15.395 | INFO     | fedpydeseq2.core.deseq2_core.deseq2_lfc_dispersions.deseq2_lfc_dispersions:run_deseq2_lfc_dispersions:161 - Compute log fold changes...
2025-05-13 08:17:05.689 | INFO     | fedpydeseq2.core.deseq2_core.deseq2_lfc_dispersions.deseq2_lfc_dispersions:run_deseq2_lfc_dispersions:171 - Finished computing log fold changes.
2025-05-13 08:17:05.689 | INFO     | fedpydeseq2.core.deseq2_core.deseq2_full_pipe:run_deseq_pipe:112 - Finished running LFC and dispersions.
2025-05-13 08:17:05.689 | INFO     | fedpydeseq2.core.deseq2_core.deseq2_full_pipe:run_deseq_pipe:114 - Computing Cook distances...
2025-05-13 08:21:28.261 | INFO     | fedpydeseq2.core.deseq2_core.deseq2_full_pipe:run_deseq_pipe:128 - Finished computing Cook distances.
2025-05-13 08:21:28.261 | INFO     | fedpydeseq2.core.deseq2_core.deseq2_full_pipe:run_deseq_pipe:132 - Refitting Cook outliers...
2025-05-13 08:23:54.524 | INFO     | fedpydeseq2.core.deseq2_core.deseq2_lfc_dispersions.deseq2_lfc_dispersions:run_deseq2_lfc_dispersions:94 - Fit genewise dispersions...
2025-05-13 08:31:38.142 | INFO     | fedpydeseq2.core.deseq2_core.deseq2_lfc_dispersions.deseq2_lfc_dispersions:run_deseq2_lfc_dispersions:108 - Finished fitting genewise dispersions.
2025-05-13 08:31:41.547 | INFO     | fedpydeseq2.core.deseq2_core.deseq2_lfc_dispersions.deseq2_lfc_dispersions:run_deseq2_lfc_dispersions:145 - Fit MAP dispersions...
2025-05-13 08:32:22.228 | INFO     | fedpydeseq2.core.deseq2_core.deseq2_lfc_dispersions.deseq2_lfc_dispersions:run_deseq2_lfc_dispersions:158 - Finished fitting MAP dispersions.
2025-05-13 08:32:22.228 | INFO     | fedpydeseq2.core.deseq2_core.deseq2_lfc_dispersions.deseq2_lfc_dispersions:run_deseq2_lfc_dispersions:161 - Compute log fold changes...
2025-05-13 08:39:52.777 | INFO     | fedpydeseq2.core.deseq2_core.deseq2_lfc_dispersions.deseq2_lfc_dispersions:run_deseq2_lfc_dispersions:171 - Finished computing log fold changes.
2025-05-13 08:39:56.449 | INFO     | fedpydeseq2.core.deseq2_core.deseq2_full_pipe:run_deseq_pipe:164 - Finished refitting Cook outliers.
2025-05-13 08:39:56.450 | INFO     | fedpydeseq2.core.deseq2_core.deseq2_full_pipe:run_deseq_pipe:168 - Running DESeq2 statistics.
2025-05-13 08:39:56.450 | INFO     | fedpydeseq2.core.deseq2_core.deseq2_stats.deseq2_stats:run_deseq2_stats:64 - Running Wald tests.
2025-05-13 08:40:09.831 | INFO     | fedpydeseq2.core.deseq2_core.deseq2_stats.deseq2_stats:run_deseq2_stats:74 - Finished running Wald tests.
2025-05-13 08:40:09.831 | INFO     | fedpydeseq2.core.deseq2_core.deseq2_stats.deseq2_stats:run_deseq2_stats:77 - Running Cook's filtering...
2025-05-13 08:40:34.250 | INFO     | fedpydeseq2.core.deseq2_core.deseq2_stats.deseq2_stats:run_deseq2_stats:86 - Finished running Cook's filtering.
2025-05-13 08:40:34.250 | INFO     | fedpydeseq2.core.deseq2_core.deseq2_stats.deseq2_stats:run_deseq2_stats:87 - Computing adjusted p-values...
2025-05-13 08:40:39.967 | INFO     | fedpydeseq2.core.deseq2_core.deseq2_stats.deseq2_stats:run_deseq2_stats:99 - Finished computing adjusted p-values.
2025-05-13 08:40:39.968 | INFO     | fedpydeseq2.core.deseq2_core.deseq2_full_pipe:run_deseq_pipe:178 - Finished running DESeq2 statistics.
2025-05-13 08:40:39.968 | INFO     | fedpydeseq2.core.deseq2_core.deseq2_full_pipe:run_deseq_pipe:182 - Saving pipeline results.
2025-05-13 08:40:43.896 | INFO     | fedpydeseq2.core.deseq2_core.deseq2_full_pipe:run_deseq_pipe:191 - Finished saving pipeline results.
2025-05-13 08:40:43,896 - INFO - Experiment summary saved to /tmp/tmp6lx_lzt9/2025_05_13_08_04_20_simu-b3e07035-3e94-4719-8802-b943363f1431.json
2025-05-13 08:40:43,896 - INFO - The compute plan has been simulated, its key is simu-b3e07035-3e94-4719-8802-b943363f1431.

Results

The results are then stored in a fl_results dictionary, which does not contain any individual sample information.

fl_results.keys()

Out:

dict_keys(['gene_names', 'MAP_dispersions', 'dispersions', 'genewise_dispersions', 'non_zero', 'fitted_dispersions', 'LFC', 'padj', 'p_values', 'wald_statistics', 'wald_se', 'replaced', 'refitted', 'prior_disp_var', '_squared_logres', 'contrast'])

We can then extract the results for our contrast of interest, and store them in a pandas DataFrame.

res_df = pd.DataFrame()
res_df["LFC"] = fl_results["LFC"]["stage_Advanced_vs_Non-advanced"]
res_df["pvalue"] = fl_results["p_values"]
res_df["padj"] = fl_results["padj"]

res_df = res_df.loc[fl_results["non_zero"], :]

display(res_df)

Out:

                      LFC    pvalue      padj
ENSG00000223972  0.338364  0.052181  0.253126
ENSG00000278267  0.038183  0.821323  0.937148
ENSG00000227232  0.104173  0.207068  0.516585
ENSG00000284332  0.174178  0.742368       NaN
ENSG00000243485  0.291516  0.192281  0.497940
...                   ...       ...       ...
ENSG00000215506 -0.585729  0.145101       NaN
ENSG00000227629  0.074826  0.933744       NaN
ENSG00000231514  0.026989  0.853410  0.948483
ENSG00000237917  0.118084  0.668620  0.867462
ENSG00000235857 -1.069862  0.247121       NaN

[57832 rows x 3 columns]

Total running time of the script: ( 38 minutes 21.225 seconds)

Download Python source code: plot_demo.py

Download Jupyter notebook: plot_demo.ipynb

Gallery generated by mkdocs-gallery