Vast Data Engine: Revolutionizing AI Data Management for 2024

Jeff Denworth, co-founder of Vast Data, introduces the Vast Data Engine, a product aimed at creating a 'thinking machine' through advanced data processing capabilities.

Denworth emphasizes three core architectural principles: adherence to standards to avoid code rewriting, simplicity in infrastructure and data management, and customer control over data and infrastructure, allowing for a personalized zero-trust agenda.

The Vast Data Engine is described as the logic behind Vast Data's distributed AI computer, designed to evolve and discover insights as AI applications advance. This initiative builds on work that began in 2016, culminating in the introduction of the Vast Data Store in 2019.

The Vast Data Store, launched in 2019, serves as a universal storage system that addresses the long-standing performance and capacity trade-offs faced by customers, particularly in managing unstructured data for AI applications.

Denworth introduces the Vast Database, the world's first transactional and analytical database that integrates both functionalities down to the archive level, establishing a semantic layer for AI computing.

The Vast Data Space is created to facilitate global data processing, allowing customers to seamlessly transition between different cloud environments, thus enhancing the overall data management experience.

The Vast Data Engine is set to be available in 2024, marking the final component of Vast Data's infrastructure aimed at simplifying AI deployment for customers at any scale.

Denworth outlines a classic data processing pipeline, which includes various stages such as data landing, transformation in data lakes, and preparation for AI computing, highlighting the complexity and multiple steps involved in managing data effectively.

The discussion begins with the observation that traditional computing methods do not align with human learning processes. The speaker emphasizes the need for a system that mirrors human learning, which involves continuous interaction with the environment and real-time data processing. This necessitates a redefinition of 'real-time' to ensure that learning occurs from events as they happen.

The speaker highlights the limitations of current data platforms, which are primarily designed for structured data and traditional analytics. These platforms typically handle data in the terabyte to petabyte range, processed by CPUs in centralized data lakes. However, deep learning requires handling vast amounts of unstructured data, such as images and text, which cannot be easily categorized or processed using existing architectures.

The speaker points out that deep learning involves processing data at an unprecedented scale, often reaching petabytes to exabytes. This necessitates the use of distributed systems that can operate globally, as traditional data lake architectures are insufficient for the gravity of such large datasets. The need for GPUs over CPUs is also emphasized due to the nature of the data being processed.

In response to the identified challenges, the speaker introduces the VAST data engine, designed for continuous and recursive computing. This system allows data to flow through it as events, triggering additional processing and correlations. Unlike traditional batch-oriented systems, the VAST data engine supports real-time processing and introduces a new AI data format that accommodates both structured and unstructured data, complete with versioning for model lineage.

The speaker explains the DAIS architecture, which is the first modern systems architecture since the introduction of the Google File System in 2003. DAIS enables a data center-scale computer with a collection of CPUs and a high-performance, low-latency network. This architecture allows for explicit parallel processing and resilience, adopting a 'cattle versus pets' mentality where infrastructure can fail without affecting overall system availability.

Customers are deploying infrastructure with six nines of availability, supporting hundreds of petabytes in a common cluster, a feat achieved with the DAIS architecture. This level of scale and availability is unprecedented, showcasing the capabilities of the system.

A core principle of the DAIS architecture is achieving efficiency, ensuring that all-flash infrastructure costs no more than traditional hard drive-based infrastructure. This is accomplished through various innovative efficiency approaches, allowing for a single tier of flash for all data.

The DAIS architecture combines state with logic at a data center scale, processing data through multiple interfaces. It supports protocols for unstructured data, query handling, stream ingestion, and notifications, alongside a serverless environment for executing built-in and customer-provided functions across numerous containers.

The system supports various data representations necessary for building a distributed AI pipeline, including events, files, objects, tables, functions, and triggers. Policies for quality of service, security, and replication can be applied from a single stateful namespace, enabling comprehensive data management without the need for additional systems.

The introduction of the VaST data engine aims to 'breathe life into data' by adding two new types of data elements: triggers and functions. These elements interact with data and metadata, creating a dynamic system where data flow can initiate triggers, which in turn can call functions to generate additional data.

The VaST data platform effectively marries structured and unstructured data, allowing files and objects to be stored in a high-scale, low-cost architecture while also supporting structured data tables with ACID transactional guarantees. This integration facilitates high-performance querying across all data types within a single system.

The new VaST Streams interface combines a streaming engine with a high-performance tabular data store, eliminating the need for separate systems for data ingestion and storage. This allows topics to be treated as tables, enabling ACID transactions and real-time querying of both new and existing data, thus transforming the streaming paradigm.

The VaST database underpins a reactive mode of computing, allowing systems to interact with the natural world at the speed of thought. This innovative approach combines transactional and analytical performance, enabling comprehensive analytics down to the archive level, marking a significant advancement in data processing capabilities.

The discussion emphasizes a shift from traditional batch computing to continuous streaming, highlighting the integration of a streaming interface with the VaST database. This innovative approach eliminates the need for separate data stores, allowing real-time queries on vast amounts of data, including exabytes, thus enabling immediate insights and interactions with the natural world.

The VaST database is presented as the world's first transactional and analytical database that maintains high transactional performance even at the archive level. This system supports continuous queries and analytics, merging structured and unstructured data, which is crucial for real-time data processing and correlation.

A built-in notification system within the VaST platform is described, which alerts users to every data update, including creates, deletes, and updates. This notification engine serves as a communication bus, facilitating interactions between various components of the system and external applications, thereby enhancing the overall functionality of AI pipelines.

The platform includes a programming environment supported by a Python SDK, allowing users to efficiently manage data. An example is provided where photos are cataloged in real-time as they enter the system, demonstrating how unstructured data is processed and stored in the VaST database, including the generation of thumbnails.

The architecture of the VaST data platform is designed for high efficiency, utilizing C++ for optimal machine utilization. This single-system approach not only enhances data and compute efficiency but also transforms how data platforms are built, promoting a real-time, continuous computing paradigm.

The event notification engine is crucial for tracking data changes and function executions within the system. This capability allows for seamless integration of internal functions and external applications, ensuring that all events are communicated effectively, thus supporting a robust programming environment capable of operating at exabyte scale.

The Daze architecture is highlighted as a web-scale flash solution that offers performance levels comparable to archive infrastructure. This architecture is foundational for the VaST data platform, enabling it to deliver high performance at a cost-effective rate, making it suitable for extensive data processing needs.

The discussion emphasizes the implementation of functions within a C++-based environment, highlighting its efficiency from a hardware perspective. This integrated system simplifies management and reduces costs, marking a shift from a batch environment to a real-time, continuous computing environment where data is central to operations.

The programming environment is described as serverless, utilizing a Python-based SDK that facilitates autoscaling of execution nodes. This allows for deployment across various locations, including the cloud, edge, and core data centers, all contributing to a unified global computing system.

The speaker introduces a novel approach to data management, where code and triggers are treated as data. This paradigm shift enables the creation of a pipeline that dynamically links code and data, allowing for seamless updates across the system whenever a function is modified.

The concept of a 'vast data set' is introduced, serving as a foundation for training models. This approach allows for the creation of materialized views around data without the need for specific file structures, as information is organized within the vast database.

A detailed description of a data processing pipeline is provided, illustrating how data flows into the system, is stored in the vast data store, and cataloged in the vast database. The pipeline includes functions for inference, data augmentation, and training model refinement, all while maintaining a structure that accommodates both structured and unstructured data.

The system's deployment capabilities are highlighted, indicating that it can operate in any environment with a vast data space. It supports various hardware configurations, including CPU and GPU, and features auto-scaling based on workload policies, showcasing its versatility and adaptability.

The speaker hints at future developments, with plans to reveal more details in 2014 and launch the product in 2024. The overarching vision is to integrate logic with state and compute with storage, enhancing the functionality of data in the context of next-generation deep learning.