We found that all analytical systems deal with the same burdens caused by data ingestion. In addition, these systems are never able to consistently communicate the data transformation rules applied to the data in terms that the business user community can understand. This inability to communicate data transformation rules results in an even greater need to rely on technical staff to address these business- driven inquiries.

SparkLoader (Extract, Load, Enhance, and Virtually Transform) is a platform-agnostic solution that leverages dynamic SQL to profile, improve, and prepare incoming data for future analytics. SparkLoader addresses common problems upfront while allowing systems to ingest self-described data without custom code. Time-versioned views are applied to implement any remaining complex business rules.

In addition to nearly eliminating source dependencies, SparkLoader also allows users to re-run their analytics against the system as it was in the past.