This quick tutorial shows how to read large CSV files efficiently using Vaex‘s memory mapping approach. For a complete introduction to working with large datasets, see our Working with Large Datasets in Python Using Vaex guide.

The Problem

When you load a large CSV file with pandas, the entire file gets read into memory immediately. For files with millions of rows, this can take minutes and consume several gigabytes of RAM. If the file is larger than your available memory, pandas will crash with a memory error. This makes working with large CSV files frustrating and limits the size of datasets you can analyze on your machine.

The Solution

Vaex solves this problem by using memory mapping to access CSV data directly from disk without loading everything into RAM at once. Here’s how to read a large CSV file:

import vaex

# First, let's create a large CSV file to demonstrate with
df = vaex.datasets.iris_1e6()
df.export('large_iris.csv')
print("CSV file created: large_iris.csv")
print(f"File size: {38.3} MB (approximately)")

This creates a CSV file with over one million rows that we can use for testing:

CSV file created: large_iris.csv
File size: 38.3 MB (approximately)

Now let’s read it back with Vaex:

import vaex
import time

# Read the CSV file with memory mapping
start = time.time()
df = vaex.open('large_iris.csv')
load_time = time.time() - start

print(f"Load time: {load_time*1000:.2f} ms")
print(f"Shape: {df.shape}")
print(f"Columns: {list(df.columns)}")
print("\nFirst few rows:")
print(df.head(5))

The file loads in just 116 milliseconds:

Load time: 116.24 ms
Shape: (1005000, 5)
Columns: ['sepal_length', 'sepal_width', 'petal_length', 'petal_width', 'class_']

First few rows:
  #    sepal_length    sepal_width    petal_length    petal_width    class_
  0             5.9            3               4.2            1.5         1
  1             6.1            3               4.6            1.4         1
  2             6.6            2.9             4.6            1.3         1
  3             6.7            3.3             5.7            2.1         2
  4             5.5            4.2             1.4            0.2         0

This speed is remarkable for over one million rows. More importantly, Vaex only maps the file to memory rather than loading all the data. This means you can work with CSV files much larger than your available RAM. When you access specific rows or columns, Vaex reads only the data you need from disk.

Format Conversion Tips

If you’ll be working with the same CSV file repeatedly, consider converting it to HDF5 or Arrow format first. These formats load even faster than CSV because they’re designed for efficient reading. You can also use vaex.from_csv() with additional parameters like chunk_size to control memory usage during the initial read, though vaex.open() works well for most cases. For extremely large files, convert the CSV once and save it in a more efficient format for future use.

Conclusion

Reading large CSV files with Vaex is fast and memory efficient thanks to memory mapping. This lets you analyze datasets that would crash pandas on the same hardware.