Data Sentinel is a powerful framework for data quality validation and monitoring in production data pipelines. It provides a comprehensive suite of tools to ensure data accuracy, completeness, consistency, and integrity with native support for PySpark and Pandas dataframes.
Data Sentinel is designed with software engineering best practices to help you create robust, maintainable, and scalable data quality monitoring solutions.
To install Data Sentinel from the Python Package Index (PyPI) run:
pip install datasentinelFor specific use cases, you can install with optional dependencies:
# For PySpark focused data validation
pip install datasentinel[pyspark-checks]
# For Pandas focused data validation
pip install datasentinel[pandas-checks]
# For complete installation with all features
pip install datasentinel[all]| Feature | What is this? |
|---|---|
| Data Quality Validations | Execute comprehensive checks to ensure data accuracy, completeness, consistency, and integrity using industry-standard validation rules. |
| Multi-DataFrame Support | Native support for PySpark and Pandas dataframes with consistent APIs. |
| Audit Stores | Comprehensive audit trail logging to multiple destinations including databases and Delta tables. |
| Notifications | Configurable notification system that alerts stakeholders when data quality issues are detected. |
| Validation Results Stores | Store data quality validation results in various formats and destinations for reporting, analysis, and historical tracking. |
Data quality is critical for successful data-driven organizations, but implementing comprehensive data quality monitoring can be complex and time-consuming. Data Sentinel addresses this by providing:
- Standardized approach to data quality validation across different technologies.
- Extensible architecture that adapts to your specific requirements.
- Best practices built-in for audit logging, notifications, and result management.
We welcome contributions to Data Sentinel! Whether you're fixing bugs, adding features, improving documentation, or sharing feedback, your contributions help make Data Sentinel better for everyone.
Check out our contribution guidelines to get started.
- Documentation: [Coming Soon] - Comprehensive guides and API reference
- GitHub Repository: https://github.com/SumzCol/datasentinel
- Issue Tracker: https://github.com/SumzCol/datasentinel/issues
Data Sentinel is licensed under the Apache Software License (Apache 2.0).