Federated Learning ECI 2026 Project

Motivation & Objectives

The goal of the project is to develop practical experience, end to end, with federated learning. You are asked to build a skeleton federated learning system by yourself, on a small dataset, and then practice with the numbers and discussions we had during the lectures.

Dataset

Choose one among the following datasets:

  1. Fashion MNIST. A dataset of Zalando's article images consisting of a training set of 60,000 examples and a test set of 10,000 examples. Each example is a 28x28 grayscale image, associated with a label from 10 classes.
  2. CIFAR10. A dataset consisting of 60000 32x32 colour images in 10 classes, with 6000 images per class. There are 50000 training images and 10000 test images.

Load the dataset, combine the default train and test splits into a single unified dataset. Split the unified dataset into a training (60%), validation (10%) and a test (30%) set using random_state=42. You must perform a stratified split to maintain identical class distributions between the sets. Verify this by printing the class distributions for all sets. Convert the target labels appropriately for multi-class classification. Carry out a data exploratory analysis by producing the following plots:

  1. A sample grid that provides 10 random samples from each of the 10 classes of the dataset.
  2. The histogram of the classes distribution.
  3. Partition the train dataset across K = 10 clients using a Dirichlet‑based label‑skew partition: for each class, draw a proportion vector from Dir(α) and allocate the class's samples to clients according to that vector (McMahan‑style Dirichlet partitioning, formalized as FedArtML's label‑skew protocol). The concentration parameter alpha controls the skew: use the following values [1000, 1, 0.5, 0.1]. For each of the resulting partitions produce the following:
    • The Hellinger Distance (HD), Earth Movers Distance (EMD), the Jensen-Shannon divergence (JSD) and the Population Stability Index (PSI). For the PSI and EMD use the global/centralized dataset as reference.
    • Plot a bubble chart of the classes distribution for each of the resulting partitions.
    • Plot a stacked bar chart of the classes distribution for each of the resulting partitions.
    • Plot the quantity skew of each client, as a natural side-effect of label partitioning, for each of the resulting partitions.

Centralized Baseline

Establish the centralized baseline using PyTorch as follows:

  1. Load the dataset, combine the default train and test splits into a single unified dataset and normalize the pixel feature values to the range [0, 1], and ensure the input shape is appropriately formatted as a 4D tensor for a 2D CNN.
  2. Build a sequential 2D CNN with the following architecture:
    • Feature extractor part:
      • A 2D convolutional layer of 8 filters of size 7 by 7
      • An Average Pooling layer of size 2 by 2
      • A 2D convolutional layer of 16 filters of size 5 by 5
      • An Average Pooling layer of size 2 by 2
      • A 2D convolutional layer of 32 filters of size 3 by 3
    All convolutional layers should maintain the spatial dimensions (height and width) using zero padding ( padding='same' ) and should have a stride of 1.
    • Classifier part:
      • A Flatten layer to transition from 2D feature maps to a 1D vector
      • A fully connected (Dense) layer with 128 units
      • A fully connected (Dense) layer with 64 units
      • The output layer
    Both hidden fully connected layers should be preceded by a Dropout layer with a 30% rate. All Convolutional and Fully Connected hidden layers should use the ReLU activation function. The output layer should have the appropriate activation function and unit count for a 10-class classification problem. After you create the model, display its summary and note the total number of trainable parameters.
  3. Compile the model with the Adam optimizer, use accuracy as the metric, and select the appropriate loss function that matches your target label format from Step 2. Write a training loop that tracks validation accuracy and stops if it hasn't improved for 3 epochs, restoring the best weights. Allow the model to train for up to 100 epochs with a batch size of 32. What is the final validation accuracy?
    • Replace the Adam optimizer with SGD (with momentum, e.g. 0.9). Allow the model to train for up to 100 epochs with a batch size of 32. What is the final validation accuracy?
  4. Plot the history of the loss and the accuracy for both the training and validation sets on a single graph. Comment on your results (e.g., check for overfitting or underfitting).
  5. Plot the confusion matrix for the test set. Which pairs of object classes tend to cause the most confusion? Calculate the test set accuracy. Was the validation set accuracy a good approximation of performance?
  6. For each test instance, calculate the model's confidence in its predicted class, defined as the maximum predicted probability from the output layer. Report the average confidence for the correctly classified test instances versus the incorrectly classified test instances. Plot the distribution of prediction confidence for correct and incorrect predictions on a histogram. Comment on whether the model tends to be less confident when it makes mistakes.

Local-only Baseline

  1. For each of the four Dirichlet‑based label‑skew partitions you created, repeat the process followed for producing the centralized baseline and train one independent model, one for each of the K clients, using only the data available to each client, with no communication at all.
  2. Allow the model to train for up to 100 epochs using the Adam and SGD as optimizers with a batch size of 32. What is the final average validation accuracy achieved among the K clients?
  3. Identify the best performing client.
    1. Plot the history of the loss and the accuracy for both the validation and training set on a single graph. Comment on your results (e.g., check for overfitting or underfitting).
    2. Plot the confusion matrix for the common/global test set. Which pairs of object classes tend to cause the most confusion? Calculate the test set accuracy. Was the validation set accuracy a good approximation of performance?
    3. For each test instance, calculate the model's confidence in its predicted class, defined as the maximum predicted probability from the output layer. Report the average confidence for the correctly classified test instances versus the incorrectly classified test instances. Plot the distribution of prediction confidence for correct and incorrect predictions on a histogram. Comment on whether the model tends to be less confident when it makes mistakes.

Federated Learning

Implement Federated Learning using the following five aggregation strategies - all five are available as built-in Flower strategies (flwr.server.strategy). For each, cite the original paper in your report and note the one or two hyperparameters that matter most:

  1. FedAvg - local epochs E, client fraction
  2. FedProx - mu (proximal coefficient)
  3. FedAvgM - Server-side momentum on the aggregated update beta
  4. FedAdam - server LR eta, tau (numerical stability), beta_1, beta_2
  5. FedPer - which layers are personalized

For FedProx, FedAvgM and FedAdam, do a small grid search (2–3 values each) rather than guessing one value. Run ≥3 random seeds per (strategy, alpha) cell and report mean ± standard deviation. A single-seed result is not evidence in FL research - every table you've seen in this course reports variance for exactly this reason.

Produce one plot (accuracy vs. HD, one line per strategy) and one summary table (rows = alpha, columns = strategy, cells = mean ± std accuracy) and answer, with evidence, not intuition:

  1. At alpha = 1000 (≈IID), do all five strategies perform similarly?
  2. As alpha decreases, which strategy degrades slowest? Does the ranking change between moderate (alpha = 0.5) and severe (alpha = 0.1) skew?
  3. Does any strategy actually improve on FedAvg, or does it only reduce variance / stabilize training without moving the mean?
  4. For FedPer specifically: compare its global-shared-layer accuracy against its personalized (per-client) accuracy.
  5. Report total bytes exchanged per strategy

How to submit

  • A public Github repository
    • The code you used to solve the above tasks
    • A technical report with all the plots and tables along with your findings.
    • Make sure that the readme file includes a link to your linkedin pages.
  • Send me an email with your Github repository link.