Building an HPC Cluster with Dask — Parallel Computing on Mini Servers

Building an HPC Cluster with Dask — Parallel Computing on Mini Servers

When 6 single-board computers become one supercomputer.

What We Built

You have 6 tiny computers — each the size of a credit card. You need to run a Monte Carlo simulation with tens of thousands of calculations. One computer? Slow. Six computers working in parallel? Roughly 6x faster.

That’s exactly what ATOMOS HPC Cluster is: a parallel computing cluster built from 6 ARM mini servers, connected over a local network, managed by Dask to distribute tasks automatically.

I built this because I wanted a cheap, practical way to run parallel Python workloads — backtesting simulations, parameter sweeps, data collection — without paying for cloud compute. Total hardware cost: roughly $200 for all six nodes.

Cluster Architecture

The setup uses a scheduler-worker topology. One node acts as the coordinator (scheduler), and the other five do the actual computation (workers). All six are on the same local network.

Totals: 24 CPU cores, 10.8GB RAM, 6 nodes

All nodes run Debian/Armbian. The scheduler exposes a web dashboard on port 8787 for real-time monitoring.

How to Set Up

1. Install Python + Dask on All Nodes

# Run on every node (221, 223, 224, 225, 226, 227)
sudo apt-get update
sudo apt-get install -y python3-pip python3-venv
python3 -m venv ~/dask-env
source ~/dask-env/bin/activate
pip install dask[complete] distributed

2. Start the Scheduler

Run this only on the scheduler node (atomos-07, 10.0.0.227):

# scheduler.py
from distributed import Scheduler
import asyncio

async def main():
    s = await Scheduler(
        port=8786,
        dashboard_address=':8787',
        address='tcp://10.0.0.227:8786'
    )
    print(f"[OK] Scheduler running at {s.address}")
    print(f"[OK] Dashboard at http://10.0.0.227:8787")
    await s.finished()

asyncio.run(main())

# Terminal
python scheduler.py &

3. Connect Workers

Run this on each of the 5 worker nodes:

# worker.py
from distributed import Worker
import asyncio

async def main():
    w = await Worker(
        'tcp://10.0.0.227:8786',
        nthreads=1,
        memory_limit='512MB',
        local_directory='/tmp/dask-worker'
    )
    print(f"[OK] Worker connected: {w.address}")
    await w.finished()

asyncio.run(main())

# Terminal (on each worker)
python worker.py &

4. Verify the Cluster

From any node:

from dask.distributed import Client

client = Client('tcp://10.0.0.227:8786')
print(client)
print(f"Workers: {len(client.scheduler_info()['workers'])}")

Expected output:

<Client: scheduler='tcp://10.0.0.227:8786' processes=5 threads=5>
Workers: 5

How to Use

Basic Example: Hello World

from dask.distributed import Client

client = Client('tcp://10.0.0.227:8786')

def square(x):
    return x ** 2

# Submit tasks to the cluster
futures = client.map(square, range(20))
results = client.gather(futures)

print(results)
# [0, 1, 4, 9, 16, 25, 36, 49, 64, 81, 100, ...]

Monte Carlo π Estimation

This is a classic parallel problem — each worker independently estimates π using random sampling:

import random
from dask.distributed import Client

client = Client('tcp://10.0.0.227:8786')

def estimate_pi_chunk(n_samples):
    """Monte Carlo sampling on a single worker."""
    random.seed()
    inside = 0
    for _ in range(n_samples):
        x = random.uniform(-1, 1)
        y = random.uniform(-1, 1)
        if x*x + y*y <= 1:
            inside += 1
    return inside

# Distribute 500,000 samples across 5 workers
n_total = 500_000
n_per_worker = n_total // 5

futures = client.submit_chunked(
    estimate_pi_chunk,
    chunks=[n_per_worker] * 5
)
results = client.gather(futures)

total_inside = sum(results)
pi_estimate = 4.0 * total_inside / n_total

print(f"π ≈ {pi_estimate:.6f}")
print(f"Error: {abs(pi_estimate - 3.141593):.6f}")

Portfolio Risk Simulation

Run 500 independent portfolio scenarios across the cluster:

import numpy as np
from dask.distributed import Client

client = Client('tcp://10.0.0.227:8786')

def simulate_portfolio(scenario_id):
    """Simulate 1 year of portfolio returns."""
    np.random.seed(scenario_id)
    daily_returns = np.random.normal(
        loc=0.0005,    # mean daily return
        scale=0.02,    # daily volatility
        size=252       # trading days
    )
    total_return = np.prod(1 + daily_returns) - 1
    max_drawdown = np.min(np.cumprod(1 + daily_returns) / 
                  np.maximum.accumulate(np.cumprod(1 + daily_returns)) - 1)
    return {
        'return': float(total_return),
        'max_drawdown': float(max_drawdown),
        'profitable': total_return > 0
    }

# Submit 500 scenarios in parallel
futures = [client.simulate_portfolio, i) for i in range(500)]
# Actually using map (much cleaner):
futures = client.map(simulate_portfolio, range(500))
results = client.gather(futures)

# Analyze
import pandas as pd
df = pd.DataFrame(results)
print(f"Mean return: {df['return'].mean():.2%}")
print(f"Median return: {df['return'].median():.2%}")
print(f"Best: {df['return'].max():.2%}")
print(f"Worst: {df['return'].min():.2%}")
print(f"Profit probability: {df['profitable'].mean():.1%}")

Performance Results

Test 1: Estimating π (Monte Carlo)

500,000 random samples split evenly across 5 workers:

The 5 workers each handled 100,000 samples independently — near-linear scaling.

Test 2: Portfolio Simulation (500 scenarios × 252 trading days)

Each scenario is independent — perfect for parallel execution. No communication overhead between workers.

Scaling Behavior

Not perfectly linear due to network overhead and the scheduler’s coordination cost, but close enough for practical use.

Server Resource Usage

During Computation

Here’s what the cluster looks like while running a heavy Monte Carlo task:

Scheduler Node (atomos-07):

CPU: [||||||||||||||||||||] 85%
RAM: [||||||||||||||-------] 1.1GB / 1.8GB
NET: ↑2.3 MB/s  ↓1.8 MB/s
Dask Process: 45MB

Worker Node (typical, e.g., atomos-01):

CPU: [||||||||||||||||||||||||] 98%
RAM: [|||||||||||||||---------] 980MB / 1.8MB
NET: ↑8.1 MB/s  ↓4.2 MB/s
Dask Worker: 180MB (includes task data)

Per-Worker Breakdown

Resource Efficiency

The cluster idles at about 15W total — roughly the same as a dim light bulb. Running at full load, each node draws around 5W.

When to Use This

Good Fit

• Backtesting trading strategies — run 100 parameter combinations simultaneously
• Monte Carlo simulations — thousands of independent scenarios
• Parameter sweeps — find optimal hyperparameters across a grid
• Parallel data collection — scrape from many sources at once
• Learning distributed computing — understand how parallel task scheduling works

Not a Fit

• LLM inference — 4-core ARM chips can’t handle large models
• Deep learning training — needs GPU and lots of RAM
• Database server — too slow for heavy I/O
• Real-time trading — network latency is too high for millisecond decisions

Dask vs MPI vs Kubernetes

For this kind of hardware (small ARM boards, 6 nodes max), the choice is clear:

Dask wins here because it’s Python-native, lightweight, and designed for exactly this kind of independent-task parallelism. Kubernetes would be overkill — it needs at least 2GB RAM just to run the control plane alone.

Next Steps

1. Set up NFS for shared storage between nodes (useful for datasets)
2. Build a REST API as a gateway to the cluster
3. Integrate with Iris — I can submit compute tasks on demand
4. Set up monitoring — the built-in Dask dashboard at port 8787 already shows task progress
5. Add GPU nodes — some ARM boards have GPU support (Orange Pi 5 Pro, Rock 5B)

Built: 2026-06-27 • By Kang Ifaz & Iris AI