As organizations advance toward production-level AI, establishing a robust evaluation framework is essential for long-term success.
The demand for more sophisticated AI solutions is growing, making Large Language Models (LLMs) an integral component of many AI strategies. They provide foundational capabilities that enhance various AI applications, driving efficiency, innovation, and improved decision-making.
However, LLMs introduce unique challenges challenges extending beyond POCs, including significant security, compliance, and regulatory risks. Effective LLM evaluations are essential to ensure AI systems function correctly, align with business goals, and remain compliant.
At Thoughtworks, our world-class AI Research Lab — enhanced by our acquisition of Watchful — focuses on programmatic interpretability for LLMs, which is vital for ensuring that AI systems can be understood and trusted.
Join us to delve into their innovative research and insights that will enhance your AI initiatives.
Melbourne
Tuesday, November 12, 2024
Work Club Olderfleet
Level 35, 477 Collins Street, Melbourne VIC 3000
Sydney
Tuesday, November 19, 2024
Work Club Barangaroo
Suite 6.01, Level 6/201 Kent St, Sydney NSW 2000
- Shayan Mohanty and John Singleton, Thoughtworks AI Lab
Who should attend?
We encourage AI executive sponsors to attend alongside technical team leads to ensure alignment between strategic vision and technical implementation.
Executives: Attend the morning session to understand how evaluations drive success in GenAI. Learn effective strategies for assessing AI models and the key components of a solid evaluation framework that facilitate the shift from proof of concept to full-scale production.
Technologists: Stay until lunch for an in-depth look at LLM evaluations, including expanded definitions, current research, and future directions that can impact your strategies. Engage with peers over lunch to exchange insights and foster connections!
Agenda
8:30am to 9:30am
9:30am to 9:45am
Andy Nolan, Director of Emerging Technologies, Thoughtworks SEAANZ
9:45am to 10:30am
Shayan Mohanty and John Singleton representing Thoughtworks AI Lab
Executives and technologists are encouraged to attend this session, which will address the importance of evaluations for GenAI, including:
Effective strategies for guiding and evaluating AI models.
Key components of an effective evaluation framework.
The role of AI evaluations in facilitating the transition from proof of concept (POC) to full-scale AI production.
10:30am to 11:15am
Executives and technologists are encouraged to attend this session. Our expert panel will explore:
- The role high-quality evaluations play in meeting business goals and compliance.
- Strategies for effective collaboration among teams to align evaluations with organizational objectives.
- Key cost considerations when running evaluations.
- How evaluations can enable effective feedback loops and reduce deployment risks.
- Considerations for LLM security, compliance, and regulatory risks.
11:15am to 11:45am
11:45am to 12:45pm
Shayan Mohanty, Head of AI Research, Thoughtworks
This session is more suited to those with a technical understanding of LLMs. Shayan will provide a deeper exploration of the latest findings and research direction of Thoughtworks' AI Lab, including:
Expanding the definition of evaluations.
Future work and potential impact.
This will be a highly interactive session, offering you the chance to ask challenging technical questions and delve into the complexities of LLM evaluation and observability.
12:45pm to 1:00pm
Andy Nolan, Director of Emerging Technologies, SEAANZ, Thoughtworks
1:00pm to 2:30pm
Speakers
Director of Emerging Technologies, Thoughtworks, SEAANZ
With over 20 years of experience, Andy specializes in applying emerging technologies like AI and computer vision to real-world challenges – developing innovative solutions for challenging environments and tightly regulated industries.
Head of AI Research, Thoughtworks AI Lab
Shayan's role is focused on bridging AI development and production. Previously CEO and Co-Founder of Watchful, he has led data engineering teams at Facebook and is a Guest Scientist at Los Alamos National Laboratory, with expertise in topics like Automata Theory and Machine Teaching.
Principal Program Manager, Thoughtworks AI Lab
John is a Principal Program Manager at Thoughtworks, bringing a wealth of experience from his previous role as Co-Founder and COO at Watchful which was recently acquired by Thoughtworks. He loves good food, playing disc golf, and foraging for mushrooms in his home of Sonoma, California.
Thoughtworks is a leading global technology consultancy that integrates strategy, design and software engineering to enable enterprises and technology disruptors across the globe to thrive as modern digital businesses.
Founded in 1993, we are over 10,500 Thoughtworkers strong across 48 offices in 19 countries. For 30+ years, we’ve delivered extraordinary impact together with our clients by helping them solve complex business problems with technology as the differentiator.