When developing applications, we often need realistic-looking user data for testing and development. Python’s Faker library includes a profile provider that generates comprehensive user profiles. While some generated distributions naturally align with real-world patterns, others might need adjustment. Let’s explore how to analyze these synthetic data distributions and improve them when necessary.

Understanding Basic Profile Generation

When generating synthetic data, we start by setting a seed value to make our results reproducible. While the specific values might change with different seeds, the methods we use to evaluate the data distribution remain valid:

from faker import Faker
fake = Faker()

# Set seed for reproducibility
Faker.seed(12345)

# Generate a sample profile
profile = fake.profile()
print("Sample Profile:")
for key, value in profile.items():
    print(f"{key}: {value}")

Each time we run this code with the same seed, we get identical results. A typical profile contains various fields like name, address, blood type, and job information. Here’s what a sample profile looks like:

Sample Profile:
job: Pension scheme manager
company: Smith, Bryan and Evans
ssn: 378-25-4428
residence: 1648 Cheryl Ferry
Joannaland, NC 96890
current_location: (Decimal('20.6274575'), Decimal('134.097523'))
blood_group: B+
website: ['http://www.hood.com/', 'https://ponce.biz/']
username: justindelgado
name: Mary Martin
sex: F
address: 638 Roman Fort
Port Victorside, TN 24857
mail: brianna82@gmail.com
birthdate: 2022-08-03

Analyzing Distribution Patterns

Let’s examine two different profile attributes: sex and blood type. This comparison will demonstrate how some synthetic distributions naturally mirror reality while others might need adjustment.

# Generate multiple profiles and analyze distributions
profiles = [fake.profile() for _ in range(1000)]

# Analyze sex distribution
sex_types = [profile['sex'] for profile in profiles]

from collections import Counter
import pandas as pd

sex_dist = pd.Series(Counter(sex_types))
sex_dist_pct = (sex_dist / len(sex_types) * 100).round(1)

print("\nGenerated Sex Distribution:")
for sex, percentage in sex_dist_pct.items():
    print(f"{sex}: {percentage}%")

The sex distribution shows a nearly balanced ratio:

Generated Sex Distribution:
M: 51.8%
F: 48.2%

Now let’s analyze the blood type distribution and compare it to US population frequencies:

blood_types = [profile['blood_group'] for profile in profiles]
blood_dist = pd.Series(Counter(blood_types))
blood_dist_pct = (blood_dist / len(blood_types) * 100).round(1)

us_dist = {
    'O+': 38.0,  # Most common in US population
    'A+': 34.0,
    'B+': 9.0,
    'AB+': 3.0,
    'O-': 7.0,
    'A-': 6.0,
    'B-': 2.0,
    'AB-': 1.0   # Rarest in US population
}

print("\nBlood Type Distribution Comparison:")
print("Type  |  Generated  |  US Population")
print("-" * 40)
for blood_type, gen_pct in blood_dist_pct.items():
    us_pct = us_dist.get(blood_type, 0)
    print(f"{blood_type:4s} |    {gen_pct:5.1f}%   |     {us_pct:5.1f}%")

The above block of code will yield the below output. When generating profile data using Faker, we notice that the distribution of blood types are fairly even, with each type occurring approximately 11-13% of the time. However, in actuality, the distribution is skewed towards ‘O+’ and ‘A+’ blood types that constitute above 70% of the population. 

Blood Type Distribution Comparison:
Type  |  Generated  |  US Population
----------------------------------------
O+   |    12.5%   |     38.0%
A+   |    11.2%   |     34.0%
B+   |    13.4%   |      9.0%
AB+  |    12.0%   |      3.0%
O-   |    13.1%   |      7.0%
A-   |    12.7%   |      6.0%
B-   |    12.0%   |      2.0%
AB-  |    13.1%   |      1.0%

This discrepancy illustrates a common challenge with synthetic data generation – default distributions often assume equal probabilities when real-world frequencies are significantly unbalanced. Such differences can impact the validity of testing scenarios, especially in healthcare or medical applications where blood type distributions matter.

Creating More Realistic Distributions

When we need more realistic distributions, we can create custom providers. The key concept here is weighted random selection – instead of giving each blood type an equal chance of being selected, we’ll assign weights that correspond to their real-world frequencies. For example, O+ will have a weight of 38 (representing 38% of the population) while AB- will have a weight of 1 (representing 1% of the population). Here’s how to implement this approach to match real-world frequencies:

from faker.providers import BaseProvider
import random

class CustomBloodTypeProvider(BaseProvider):
    blood_types_with_weights = {
        'O+': 38, 'A+': 34, 'B+': 9, 'AB+': 3,
        'O-': 7, 'A-': 6, 'B-': 2, 'AB-': 1
    }
    
    def realistic_blood_type(self):
        return random.choices(
            list(self.blood_types_with_weights.keys()),
            weights=list(self.blood_types_with_weights.values()),
            k=1
        )[0]

# Register and use the custom provider
fake.add_provider(CustomBloodTypeProvider)

# Test the custom provider
test_blood_types = [fake.realistic_blood_type() for _ in range(1000)]
test_dist = pd.Series(Counter(test_blood_types))
test_dist_pct = (test_dist / len(test_blood_types) * 100).round(1)

print("\nCustom Provider Blood Type Distribution:")
print("Type  |  Generated  |  Expected")
print("-" * 40)
for blood_type in sorted(test_dist_pct.index):
    expected = CustomBloodTypeProvider.blood_types_with_weights[blood_type]
    print(f"{blood_type:4s} |    {test_dist_pct[blood_type]:5.1f}%   |     {expected:5.1f}%")

Let’s test our custom provider to see the improved distribution:

Custom Provider Blood Type Distribution:
Type  |  Generated  |  Expected
----------------------------------------
A+   |    35.6%   |     34.0%
A-   |     6.5%   |      6.0%
AB+  |     2.6%   |      3.0%
AB-  |     0.9%   |      1.0%
B+   |     7.6%   |      9.0%
B-   |     2.0%   |      2.0%
O+   |    38.9%   |     38.0%
O-   |     5.9%   |      7.0%

As we can see from the results, our custom provider generates blood type frequencies that closely mirror real-world distributions, with O+ and A+ appearing most frequently and rare types like AB- occurring at their natural low frequencies. This improved distribution makes our synthetic data more suitable for testing scenarios where realistic blood type frequencies are important, such as healthcare applications or medical database testing. Custom providers like this can be created for any field where specific distribution patterns need to be maintained.

Key Insights

This analysis reveals important considerations when using synthetic data:

• Some Faker distributions naturally match real-world patterns (like the sex distribution)
• Other distributions might need adjustment for realistic testing (like blood types)
• Setting a seed ensures reproducible results, essential for testing
• Custom providers offer a way to generate data matching specific distribution requirements

When using synthetic data for testing or development, take time to analyze the distributions of different fields and determine which ones need adjustment for your specific use case. Not all fields will require modification – some of Faker’s default distributions might already perfectly suit your needs. Understanding these patterns and making informed decisions about which distributions to customize can significantly improve the quality of your testing data. The key is to balance the effort of creating custom providers against the importance of realistic data distributions for your specific application.