The Transformative Power of AI: Insights from Andrew at Snowflake

Andrew expresses excitement about the current era for builders, particularly in the realm of AI, which he likens to electricity as a general-purpose technology. He emphasizes that AI is creating vast opportunities for new applications that were previously impossible.

Andrew outlines the AI stack, starting with semiconductors at the base, followed by cloud infrastructure, including Snowflake, and then foundation model trainers and models. He notes that while there is significant media hype around these technology layers, the application layer is crucial for generating value and revenue, which in turn supports the technology providers below.

The advent of generative AI has accelerated machine learning model development, allowing for faster creation of applications. Andrew illustrates this with the example of sentiment analysis, where traditional workflows could take six to twelve months, but with generative AI, prototypes can be developed in days, significantly enhancing the speed of experimentation and product development.

Andrew highlights a shift towards fast experimentation as a new path for innovation. He describes a design pattern where AI teams can quickly prototype multiple ideas over a weekend, allowing them to test numerous concepts and focus on the successful ones, rather than spending months on a single project.

Andrew points out that evaluations (or 'evals') are becoming a bottleneck in the development process. In the past, collecting additional data points for testing was manageable, but with large language model-based applications, the need for extensive testing data can significantly slow down development, creating challenges in the new workflow.

The speaker discusses the shift towards collecting data in parallel rather than sequentially, emphasizing the importance of building prototypes quickly. As the need for robustness and reliability increases, the testing process is gradually enhanced. While machine learning prototyping has accelerated, the overall software application development process remains complex, involving multiple steps such as design, integration, and deployment. The speaker notes that although some aspects of development have sped up, they have not kept pace with the rapid advancements in machine learning modeling, creating pressure on organizations to expedite all related processes.

The speaker critiques the mantra 'move fast and break things,' suggesting it has garnered a negative reputation due to its consequences. Instead, they advocate for a more responsible approach: 'move fast and be responsible.' They highlight that many teams are now capable of rapidly prototyping and testing their ideas without releasing potentially harmful products to the public. This responsible speed in development is seen as exhilarating, allowing for quicker and safer innovation in technology.

The speaker identifies agentic AI workflows as the most significant trend in AI technology, expressing excitement about its potential. Initially a controversial statement, the term 'AI agents' has gained traction among both technical and non-technical audiences. The speaker explains that traditional use of large language models often involves 'zero shot prompting,' which is akin to asking an AI to write an essay in one go. However, they propose that an agentic workflow, which includes iterative processes like outlining, research, drafting, and revising, leads to superior outcomes. This method, while more time-consuming, enhances the quality of work significantly.

The speaker shares practical applications of agentic workflows in various fields, such as processing complex legal documents, assisting in healthcare diagnoses, and ensuring compliance with government regulations. They emphasize that this iterative approach yields better results than traditional methods, showcasing the effectiveness of agentic AI in tackling intricate tasks.

Agentic workflows are being utilized to process image and video data, showing significant improvements in performance metrics. The Human Eval Benchmark, which assesses the ability of OpenAI's models to solve coding puzzles, indicates that GPT-3.5 achieved a 48% success rate, while GPT-4 improved to 67%. However, the most remarkable enhancement is observed when employing agentic workflows, where GPT-3.5 can reach up to 95% accuracy, demonstrating the effectiveness of these workflows in enhancing model performance.

The discussion identifies four major design patterns in agentic workflows: reflection, tool use, planning, and multi-agent collaboration. These patterns help demystify how agentic reasoning is implemented in applications. The speaker emphasizes that understanding the code behind these workflows reveals their simplicity and effectiveness.

The reflection design pattern involves prompting a language model (LM) to generate code for specific tasks, followed by critiquing that code. The process includes taking the generated code, asking the LM to examine and critique it, and then using that feedback to improve the code. This iterative process can significantly enhance baseline performance, showcasing how self-critique can lead to better outcomes.

The tool use design pattern allows language models to generate requests for API calls, enabling them to perform tasks such as searching the web, issuing refunds, or sending emails. This capability expands the functionality of agentic workflows, allowing for more complex interactions and tasks to be executed effectively.

In the planning design pattern, a language model can handle complex requests by determining a sequence of actions to achieve a goal. For instance, when tasked with generating an image of a girl reading a book, the LM can first analyze the scene using an open pose model, then create the image, describe it, and finally generate audio output. This structured approach to task execution highlights the model's reasoning capabilities.

The multi-agent collaboration design pattern involves prompting different language models to take on various roles, such as a coder and a critic, within the same conversation. This collaborative approach can lead to improved performance by allowing the models to provide feedback and suggestions to each other, enhancing the overall output quality.

The discussion highlights the effectiveness of prompting a language model (LM) to assume different roles at various times, allowing multiple agents to interact and collaborate on tasks. This design pattern has shown significant improvements in performance across various tasks, as evidenced by numerous teams. The analogy is drawn to CPU processes, where breaking down a task into subtasks and employing multiple agents can enhance the development of complex systems, leading to better outcomes.

Excitement is expressed regarding the emergence of large multimodal model-based agents, which can perform zero-shot predictions. For instance, these agents can analyze an image and provide outputs, such as counting runners in a race. However, it is noted that these agents perform better with iterative workflows, allowing for step-by-step problem-solving, such as detecting faces and numbers before compiling results. This iterative approach enables agents to plan, test, and write code, ultimately delivering more complex solutions.

A demonstration is presented involving a visual AI task where the goal is to count players in a soccer game image. The process initiates a complex coding task to accurately count players on the field, excluding those in the background. After running the process, it is revealed that the model successfully identified seven players. The output includes generated Python code that can be reused for analyzing a large collection of images, showcasing the potential for businesses with extensive visual data to derive value from their assets.

The discussion highlights the transformative potential of visual AI capabilities, particularly through the use of a vision agent that allows users to extract valuable insights from previously stored data. This transformation is exemplified by a scenario where a video of a soccer game is analyzed to identify and extract clips of goals being scored, showcasing the agent's ability to process video data efficiently.

Further examples illustrate the vision agent's functionality, such as splitting a video into 6-second chunks, describing each segment, and organizing the information into a Pandas DataFrame. This capability enables users to generate metadata for video content, which can be stored in platforms like Snowflake for further application development. The speaker emphasizes the ease of generating code through the vision agent, which can be utilized for various applications.

A demo application developed by the speaker's team at Landing AI showcases the practical use of the vision agent. The application allows users to search for specific video clips, such as those featuring a 'skier airborne.' The demo highlights the agent's ability to index videos and display clips with high similarity, marked in green on the timeline, thus enhancing the user experience in browsing video collections.

An engaging example is presented regarding the identification of luggage, specifically black luggage with a rainbow strap. The speaker notes the challenge of distinguishing between similar items, emphasizing the fun and utility of using visual AI to enhance the search and identification process in real-world scenarios.

The speaker discusses emerging opportunities in AI, particularly focusing on agentic workflows and the evolving AI stack. They introduce the concept of a new agentic orchestration layer, which simplifies the development of applications. The speaker expresses hope that Landing AI's vision agent will contribute to making it easier for developers to create visual AI applications that effectively process image and video data, which has historically been challenging to leverage.

The speaker identifies four significant trends in AI, emphasizing that while many developments exist, agentic AI stands out as the most crucial. The first trend involves advancements in agentic workflows, which require extensive reading of text and images, leading to increased token generation. Efforts to enhance token generation speed are underway, particularly through semiconductor innovations by companies like Sova Service and others, alongside various software and hardware improvements.

The second trend highlights the evolution of large language models (LLMs), which have traditionally been optimized to respond to human-generated questions. Recently, these models have begun to be fine-tuned for tool use, exemplified by Anthropic's recent release of a model designed to support computer use. This shift is expected to significantly enhance the capabilities of agentic workflows, allowing LLMs to perform more complex tasks beyond merely answering queries.

The third trend points to the rising importance of data engineering, particularly concerning unstructured data. The speaker notes that while machine learning has historically focused on structured data, advancements in processing capabilities for text, images, video, and potentially audio are increasing the demand for effective management of unstructured data. This includes the need for robust metadata management and deployment strategies to maximize the value derived from such data.

The fourth trend discusses the ongoing revolution in text processing, which has already made significant strides, while the image processing revolution is still in its early stages. The speaker expresses excitement about the potential for businesses to extract greater value from visual data, which could lead to a broader range of applications than ever before. This period is described as an opportune time for builders, as generative AI is facilitating faster experimentation and expanding the possibilities for new applications in both visual and non-visual AI.

To conclude, the speaker invites the audience to explore visual AI demos available at va.landing.ai, where they can interact with the demonstrations and access the underlying code for their own applications. The session wraps up with a call to welcome Elsa back to the stage, marking the end of the discussion.