Skip to content

Performance: Add lazy loading for rarely used generated models #36

@knowlen

Description

@knowlen

Description

As identified in PR #32 review, the large number of generated models could benefit from lazy loading to reduce memory usage, especially for users who only use a subset of the API.

Current State

  • All generated models are imported at module initialization
  • Memory footprint includes all models even if unused
  • Impacts startup time and memory usage

Proposed Solution

Implement lazy loading using import hooks and a registry pattern:

class LazyModelLoader:
    def __init__(self):
        self._models = {}
        self._loaded = set()
    
    def __getattr__(self, name):
        if name not in self._loaded:
            # Dynamically import the model
            module = importlib.import_module(f'esologs._generated.{name}')
            self._models[name] = module
            self._loaded.add(name)
        return self._models[name]

# Usage
models = LazyModelLoader()
# Model only loaded when accessed
character_data = models.get_character_by_id

Implementation Ideas

  1. Use Python's import hooks for transparent lazy loading
  2. Track which models are commonly used together
  3. Provide option to preload common models
  4. Consider using __init__.py with __getattr__ for module-level lazy loading

Benefits

  • Reduced memory footprint for applications using subset of API
  • Faster initial import time
  • No change to public API

Performance Metrics to Track

  • Import time before/after
  • Memory usage for minimal usage patterns
  • Memory usage for full API usage

Considerations

  • Ensure thread safety for lazy loading
  • Consider impact on type checking and IDE support
  • May need to update documentation about import behavior

References

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions