FedPyDESeq2 demo on the TCGA-LUAD dataset.

Note

Click here to download the full example code

This example demonstrates how to run a FedPyDESeq2 experiment on the TCGA-LUAD dataset from a single machine, using Substra's simulation mode.

We will show how to perform a simple differential expression analysis, comparing samples with "Advanced" vs "Non-advanced" tumoral stage.

from pathlib import Path

import pandas as pd
from fedpydeseq2_datasets.process_and_split_data import setup_tcga_dataset
from IPython.display import display

from fedpydeseq2.fedpydeseq2_pipeline import run_fedpydeseq2_experiment

Dataset setup

In a real federated setup, the data is distributed across multiple medical centers and must be registered with Substra beforehand. Hence, each center would have a folder containing two csvs (one fore the counts and one for the metadata), as well as an opener python file and a markdown readme file (see Substra's documentation on how to register a datasample). Then, we would only need pass the dataset_datasample_keys path.

In this tutorial, however, we use FedPyDESeq2's simulation mode, which allows us to emulate a federated setup from a single machine.

The simulation mode assumes the data to be organized in the following structure:

processed_data_path/
├── centers_data/
│   └── tcga/
│       └── {experiment_id}/
│           ├── center_0/
│           │   ├── counts.csv
│           │   └── metadata.csv
│           ├── center_1/
│           │   ├── counts.csv
│           │   └── metadata.csv
│           └── ...
└── pooled_data/
    └── tcga/
        └── {experiment_id}/
            ├── counts.csv
            └── metadata.csv

In this tutorial, we have already downloaded the data in the data/raw directory.

The setup_tcga_dataset function from fedpydeseq2_datasets will automatically organize the data in the data/processed directory.

It will split the TCGA-LUAD dataset into 7 centers according to the geographical origin of the samples, as described in the FedPyDESeq2 paper.

See also the fedpydeseq2_datasets repository for more details.

dataset_name = "TCGA-LUAD"
raw_data_path = Path("data/raw").resolve()
processed_data_path = Path("data/processed").resolve()
design_factors = "stage"

setup_tcga_dataset(
    raw_data_path,
    processed_data_path,
    dataset_name=dataset_name,
    small_samples=False,
    small_genes=False,
    only_two_centers=False,
    design_factors=design_factors,
    force=True,
)

experiment_id = "TCGA-LUAD-stage"

Out:

2025-02-12 09:36:04.806 | INFO     | fedpydeseq2_datasets.process_and_split_data:setup_tcga_dataset:144 - Setting up TCGA dataset: TCGA-LUAD-stage
2025-02-12 09:36:04.807 | INFO     | fedpydeseq2_datasets.process_and_split_data:setup_tcga_dataset:150 - First center metadata does not exist or force=True. Setting up the dataset.
2025-02-12 09:36:04.807 | INFO     | fedpydeseq2_datasets.process_and_split_data:_setup_tcga_dataset:282 - Processing the data for the TCGA dataset: TCGA-LUAD-stage
2025-02-12 09:36:07.746 | INFO     | fedpydeseq2_datasets.process_and_split_data:_setup_tcga_dataset:302 - Saving the data for each center /home/runner/work/fedpydeseq2/fedpydeseq2/docs/examples/data/processed/centers_data/tcga/TCGA-LUAD-stage
2025-02-12 09:36:45.787 | INFO     | fedpydeseq2_datasets.process_and_split_data:_setup_tcga_dataset:364 - Saving the pooled data at /home/runner/work/fedpydeseq2/fedpydeseq2/docs/examples/data/processed/pooled_data/tcga/TCGA-LUAD-stage

Running the experiment

We can now run the experiment.

Substra, the FL framework on which FedPyDESeq2 is built, supports a simulated mode which may be run locally from a single machine, which we will use here.

Let's run our FedPyDESeq2 experiment. This may be done using the run_fedpydeseq2_experiment wrapper function, which takes the following parameters:

n_centers=7: Our data is distributed across 7 different medical centers
backend="subprocess" and simulate=True: We'll run the analysis locally on our machine to simulate a federated setup, rather than in a real distributed environment
register_data=True: We'll register our dataset with Substra before analysis. In the case of a real federated setup, this would be set to False if data was already registered by Substra.
asset_directory: This directory should contain an opener.py file, containing an Opener class, and datasets.description.md file. Here, we copy them from fedpydeseq2_datasets/assets/tcga
centers_root_directory: Where the processed data for each center is stored
compute_plan_name: We'll call this analysis "Example-TCGA-LUAD-pipeline" in Substra
dataset_name: We're working with the TCGA-LUAD lung cancer dataset
design_factors: This should be a list of the design factors we wish to include in our analysis. Here, we're studying how "stage" (the cancer stage) affects gene expression
ref_levels: We're setting "Non-advanced" as our baseline cancer stage
contrast: This should be a list of three strings, of the form ["factor", "alternative_level", "baseline_level"]. To compare gene expression between "Advanced" vs "Non-advanced" stages, we set contrast=["stage", "Advanced", "Non-advanced"].
refit_cooks=True: After finding outliers using Cook's distance, we'll refit the model without them for more robust results

fl_results = run_fedpydeseq2_experiment(
    n_centers=7,
    backend="subprocess",
    simulate=True,
    register_data=True,
    asset_directory=Path("assets/tcga").resolve(),
    centers_root_directory=processed_data_path
    / "centers_data"
    / "tcga"
    / experiment_id,
    compute_plan_name="Example-TCGA-LUAD-pipeline",
    dataset_name="TCGA-LUAD",
    design_factors="stage",
    ref_levels={"stage": "Non-advanced"},
    contrast=["stage", "Advanced", "Non-advanced"],
    refit_cooks=True,
)

Out:

2025-02-12 09:37:23.595 | INFO     | fedpydeseq2.substra_utils.federated_experiment:run_federated_experiment:182 - Setting up organizations...
2025-02-12 09:37:23.598 | INFO     | fedpydeseq2.substra_utils.federated_experiment:run_federated_experiment:233 - Registering the datasets...
2025-02-12 09:37:23.598 | INFO     | fedpydeseq2.substra_utils.federated_experiment:run_federated_experiment:290 - Adding dataset to client MyOrg2MSP
2025-02-12 09:37:23.599 | INFO     | fedpydeseq2.substra_utils.federated_experiment:run_federated_experiment:295 - Dataset added. Key: 23bbb07e-fc60-47f5-89d3-c8e545dbb6b2
2025-02-12 09:37:23.599 | INFO     | fedpydeseq2.substra_utils.federated_experiment:run_federated_experiment:290 - Adding dataset to client MyOrg3MSP
2025-02-12 09:37:23.600 | INFO     | fedpydeseq2.substra_utils.federated_experiment:run_federated_experiment:295 - Dataset added. Key: 9f1d59b2-63f4-436c-b2fe-b9437e6be11a
2025-02-12 09:37:23.600 | INFO     | fedpydeseq2.substra_utils.federated_experiment:run_federated_experiment:290 - Adding dataset to client MyOrg4MSP
2025-02-12 09:37:23.601 | INFO     | fedpydeseq2.substra_utils.federated_experiment:run_federated_experiment:295 - Dataset added. Key: 9ce482e1-d2a1-4967-9c32-92c397336b9a
2025-02-12 09:37:23.601 | INFO     | fedpydeseq2.substra_utils.federated_experiment:run_federated_experiment:290 - Adding dataset to client MyOrg5MSP
2025-02-12 09:37:23.601 | INFO     | fedpydeseq2.substra_utils.federated_experiment:run_federated_experiment:295 - Dataset added. Key: 459bb9d3-459d-463e-8ab1-16d9f398787b
2025-02-12 09:37:23.602 | INFO     | fedpydeseq2.substra_utils.federated_experiment:run_federated_experiment:290 - Adding dataset to client MyOrg6MSP
2025-02-12 09:37:23.602 | INFO     | fedpydeseq2.substra_utils.federated_experiment:run_federated_experiment:295 - Dataset added. Key: 957075d6-6df0-47b8-b2e0-fe6053c21f81
2025-02-12 09:37:23.602 | INFO     | fedpydeseq2.substra_utils.federated_experiment:run_federated_experiment:290 - Adding dataset to client MyOrg7MSP
2025-02-12 09:37:23.603 | INFO     | fedpydeseq2.substra_utils.federated_experiment:run_federated_experiment:295 - Dataset added. Key: b7291019-4c67-4a39-ab11-4b771921814f
2025-02-12 09:37:23.603 | INFO     | fedpydeseq2.substra_utils.federated_experiment:run_federated_experiment:290 - Adding dataset to client MyOrg8MSP
2025-02-12 09:37:23.604 | INFO     | fedpydeseq2.substra_utils.federated_experiment:run_federated_experiment:295 - Dataset added. Key: e44981d1-3735-4e45-b1be-353ae30aa517
2025-02-12 09:37:23.604 | INFO     | fedpydeseq2.substra_utils.federated_experiment:run_federated_experiment:316 - Datasets registered.
2025-02-12 09:37:23.604 | INFO     | fedpydeseq2.substra_utils.federated_experiment:run_federated_experiment:318 - Dataset keys: {'MyOrg2MSP': '23bbb07e-fc60-47f5-89d3-c8e545dbb6b2', 'MyOrg3MSP': '9f1d59b2-63f4-436c-b2fe-b9437e6be11a', 'MyOrg4MSP': '9ce482e1-d2a1-4967-9c32-92c397336b9a', 'MyOrg5MSP': '459bb9d3-459d-463e-8ab1-16d9f398787b', 'MyOrg6MSP': '957075d6-6df0-47b8-b2e0-fe6053c21f81', 'MyOrg7MSP': 'b7291019-4c67-4a39-ab11-4b771921814f', 'MyOrg8MSP': 'e44981d1-3735-4e45-b1be-353ae30aa517'}
2025-02-12 09:37:47,780 - INFO - Simulating the execution of the compute plan.
2025-02-12 09:37:47.780 | INFO     | fedpydeseq2.core.deseq2_core.deseq2_full_pipe:run_deseq_pipe:66 - Building design matrices...
2025-02-12 09:37:48.206 | INFO     | fedpydeseq2.core.deseq2_core.deseq2_full_pipe:run_deseq_pipe:76 - Finished building design matrices.
2025-02-12 09:37:48.206 | INFO     | fedpydeseq2.core.deseq2_core.deseq2_full_pipe:run_deseq_pipe:82 - Computing size factors...
2025-02-12 09:37:48.629 | INFO     | fedpydeseq2.core.deseq2_core.deseq2_full_pipe:run_deseq_pipe:97 - Finished computing size factors.
2025-02-12 09:37:48.629 | INFO     | fedpydeseq2.core.deseq2_core.deseq2_full_pipe:run_deseq_pipe:101 - Running LFC and dispersions.
2025-02-12 09:37:48.629 | INFO     | fedpydeseq2.core.deseq2_core.deseq2_lfc_dispersions.deseq2_lfc_dispersions:run_deseq2_lfc_dispersions:94 - Fit genewise dispersions...
2025-02-12 09:42:54.490 | INFO     | fedpydeseq2.core.deseq2_core.deseq2_lfc_dispersions.deseq2_lfc_dispersions:run_deseq2_lfc_dispersions:108 - Finished fitting genewise dispersions.
2025-02-12 09:42:54.490 | INFO     | fedpydeseq2.core.deseq2_core.deseq2_lfc_dispersions.deseq2_lfc_dispersions:run_deseq2_lfc_dispersions:112 - Compute dispersion prior...
2025-02-12 09:42:57.582 | INFO     | fedpydeseq2.core.deseq2_core.deseq2_lfc_dispersions.deseq2_lfc_dispersions:run_deseq2_lfc_dispersions:125 - Finished computing dispersion prior.
2025-02-12 09:42:57.583 | INFO     | fedpydeseq2.core.deseq2_core.deseq2_lfc_dispersions.deseq2_lfc_dispersions:run_deseq2_lfc_dispersions:145 - Fit MAP dispersions...
2025-02-12 09:46:39.063 | INFO     | fedpydeseq2.core.deseq2_core.deseq2_lfc_dispersions.deseq2_lfc_dispersions:run_deseq2_lfc_dispersions:158 - Finished fitting MAP dispersions.
2025-02-12 09:46:39.063 | INFO     | fedpydeseq2.core.deseq2_core.deseq2_lfc_dispersions.deseq2_lfc_dispersions:run_deseq2_lfc_dispersions:161 - Compute log fold changes...
2025-02-12 09:50:27.917 | INFO     | fedpydeseq2.core.deseq2_core.deseq2_lfc_dispersions.deseq2_lfc_dispersions:run_deseq2_lfc_dispersions:171 - Finished computing log fold changes.
2025-02-12 09:50:27.917 | INFO     | fedpydeseq2.core.deseq2_core.deseq2_full_pipe:run_deseq_pipe:112 - Finished running LFC and dispersions.
2025-02-12 09:50:27.917 | INFO     | fedpydeseq2.core.deseq2_core.deseq2_full_pipe:run_deseq_pipe:114 - Computing Cook distances...
2025-02-12 09:54:49.562 | INFO     | fedpydeseq2.core.deseq2_core.deseq2_full_pipe:run_deseq_pipe:128 - Finished computing Cook distances.
2025-02-12 09:54:49.562 | INFO     | fedpydeseq2.core.deseq2_core.deseq2_full_pipe:run_deseq_pipe:132 - Refitting Cook outliers...
2025-02-12 09:57:14.943 | INFO     | fedpydeseq2.core.deseq2_core.deseq2_lfc_dispersions.deseq2_lfc_dispersions:run_deseq2_lfc_dispersions:94 - Fit genewise dispersions...
2025-02-12 10:04:50.956 | INFO     | fedpydeseq2.core.deseq2_core.deseq2_lfc_dispersions.deseq2_lfc_dispersions:run_deseq2_lfc_dispersions:108 - Finished fitting genewise dispersions.
2025-02-12 10:04:54.376 | INFO     | fedpydeseq2.core.deseq2_core.deseq2_lfc_dispersions.deseq2_lfc_dispersions:run_deseq2_lfc_dispersions:145 - Fit MAP dispersions...
2025-02-12 10:05:35.158 | INFO     | fedpydeseq2.core.deseq2_core.deseq2_lfc_dispersions.deseq2_lfc_dispersions:run_deseq2_lfc_dispersions:158 - Finished fitting MAP dispersions.
2025-02-12 10:05:35.158 | INFO     | fedpydeseq2.core.deseq2_core.deseq2_lfc_dispersions.deseq2_lfc_dispersions:run_deseq2_lfc_dispersions:161 - Compute log fold changes...
2025-02-12 10:12:59.992 | INFO     | fedpydeseq2.core.deseq2_core.deseq2_lfc_dispersions.deseq2_lfc_dispersions:run_deseq2_lfc_dispersions:171 - Finished computing log fold changes.
2025-02-12 10:13:03.754 | INFO     | fedpydeseq2.core.deseq2_core.deseq2_full_pipe:run_deseq_pipe:164 - Finished refitting Cook outliers.
2025-02-12 10:13:03.754 | INFO     | fedpydeseq2.core.deseq2_core.deseq2_full_pipe:run_deseq_pipe:168 - Running DESeq2 statistics.
2025-02-12 10:13:03.754 | INFO     | fedpydeseq2.core.deseq2_core.deseq2_stats.deseq2_stats:run_deseq2_stats:64 - Running Wald tests.
2025-02-12 10:13:16.978 | INFO     | fedpydeseq2.core.deseq2_core.deseq2_stats.deseq2_stats:run_deseq2_stats:74 - Finished running Wald tests.
2025-02-12 10:13:16.978 | INFO     | fedpydeseq2.core.deseq2_core.deseq2_stats.deseq2_stats:run_deseq2_stats:77 - Running Cook's filtering...
2025-02-12 10:13:41.034 | INFO     | fedpydeseq2.core.deseq2_core.deseq2_stats.deseq2_stats:run_deseq2_stats:86 - Finished running Cook's filtering.
2025-02-12 10:13:41.034 | INFO     | fedpydeseq2.core.deseq2_core.deseq2_stats.deseq2_stats:run_deseq2_stats:87 - Computing adjusted p-values...
2025-02-12 10:13:46.659 | INFO     | fedpydeseq2.core.deseq2_core.deseq2_stats.deseq2_stats:run_deseq2_stats:99 - Finished computing adjusted p-values.
2025-02-12 10:13:46.659 | INFO     | fedpydeseq2.core.deseq2_core.deseq2_full_pipe:run_deseq_pipe:178 - Finished running DESeq2 statistics.
2025-02-12 10:13:46.659 | INFO     | fedpydeseq2.core.deseq2_core.deseq2_full_pipe:run_deseq_pipe:182 - Saving pipeline results.
2025-02-12 10:13:50.449 | INFO     | fedpydeseq2.core.deseq2_core.deseq2_full_pipe:run_deseq_pipe:191 - Finished saving pipeline results.
2025-02-12 10:13:50,450 - INFO - Experiment summary saved to /tmp/tmp0_7jsizk/2025_02_12_09_37_47_simu-49a876ca-4c90-4274-99c8-6cd47b03aa2d.json
2025-02-12 10:13:50,450 - INFO - The compute plan has been simulated, its key is simu-49a876ca-4c90-4274-99c8-6cd47b03aa2d.

Results

The results are then stored in a fl_results dictionary, which does not contain any individual sample information.

fl_results.keys()

Out:

dict_keys(['gene_names', 'MAP_dispersions', 'dispersions', 'genewise_dispersions', 'non_zero', 'fitted_dispersions', 'LFC', 'padj', 'p_values', 'wald_statistics', 'wald_se', 'replaced', 'refitted', 'prior_disp_var', '_squared_logres', 'contrast'])

We can then extract the results for our contrast of interest, and store them in a pandas DataFrame.

res_df = pd.DataFrame()
res_df["LFC"] = fl_results["LFC"]["stage_Advanced_vs_Non-advanced"]
res_df["pvalue"] = fl_results["p_values"]
res_df["padj"] = fl_results["padj"]

res_df = res_df.loc[fl_results["non_zero"], :]

display(res_df)

Out:

                      LFC    pvalue      padj
ENSG00000223972  0.338364  0.052181  0.253126
ENSG00000278267  0.038183  0.821323  0.937148
ENSG00000227232  0.104173  0.207068  0.516585
ENSG00000284332  0.174178  0.742368       NaN
ENSG00000243485  0.291516  0.192281  0.497940
...                   ...       ...       ...
ENSG00000215506 -0.585729  0.145101       NaN
ENSG00000227629  0.074826  0.933744       NaN
ENSG00000231514  0.026989  0.853410  0.948483
ENSG00000237917  0.118084  0.668620  0.867462
ENSG00000235857 -1.069862  0.247121       NaN

[57832 rows x 3 columns]

Total running time of the script: ( 37 minutes 56.015 seconds)

Download Python source code: plot_demo.py

Download Jupyter notebook: plot_demo.ipynb

Gallery generated by mkdocs-gallery