This quick tutorial shows how to export Vaex DataFrames to HDF5 format for faster loading in future sessions. For a complete introduction to working with large datasets, see our Working with Large Datasets in Python Using Vaex guide.

The Problem

After spending time cleaning and preparing a large dataset, you don’t want to repeat that work every time you analyze the data. CSV files are convenient for sharing but slow to load repeatedly. If you’re working with the same dataset across multiple sessions, loading from CSV becomes a bottleneck. You need a format that preserves your data efficiently and loads much faster than CSV.

The Solution

Vaex can export DataFrames to HDF5 format, which loads significantly faster than CSV. Here’s how to export your data:

import vaex

# Load a dataset and export to HDF5 format
df = vaex.datasets.iris_1e6()
df.export('large_iris.hdf5')

print("HDF5 file created: large_iris.hdf5")
print(f"Dataset shape: {df.shape}")
print(f"Columns: {list(df.columns)}")

The export creates an HDF5 file ready for future use:

HDF5 file created: large_iris.hdf5
Dataset shape: (1005000, 5)
Columns: ['sepal_length', 'sepal_width', 'petal_length', 'petal_width', 'class_']

Now let’s compare loading times between CSV and HDF5:

import vaex
import time

# Load CSV file
start = time.time()
df_csv = vaex.open('large_iris.csv')
csv_time = time.time() - start

# Load HDF5 file
start = time.time()
df_hdf5 = vaex.open('large_iris.hdf5')
hdf5_time = time.time() - start

print(f"CSV load time: {csv_time*1000:.2f} ms")
print(f"HDF5 load time: {hdf5_time*1000:.2f} ms")
print(f"Speed improvement: {csv_time/hdf5_time:.1f}x faster with HDF5")

The difference is dramatic:

CSV load time: 108.31 ms
HDF5 load time: 2.41 ms
Speed improvement: 45.0x faster with HDF5

HDF5 loads 45 times faster than CSV for this dataset. For larger datasets, this difference becomes even more significant. Once you’ve prepared your data and created any virtual columns, export to HDF5 so future sessions start instantly.

When to Use HDF5

Use HDF5 when you’ll access the same dataset multiple times. After initial data cleaning, export to HDF5 before starting your analysis work. HDF5 is also excellent for archiving processed datasets or sharing data with colleagues who use Vaex. Note that virtual columns you create are stored as expressions in HDF5, so they remain virtual rather than taking up space. If you need to share data with tools that don’t support HDF5, consider Parquet format as an alternative that also loads faster than CSV.

Conclusion

Exporting to HDF5 format saves significant time when working with the same dataset repeatedly. The 45x speed improvement makes it worthwhile for any dataset you’ll use more than once.