Enable javascript in your browser for better experience. Need to know to enable it? Go here.

AI POC to production: Advanced evaluation and observability for LLMs

Join us for an engaging session with Thoughtworks' leading AI experts as they visit Australia. Discover groundbreaking insights into evaluation methods and tools for programmatic and mechanistic interpretability in LLM applications, and connect with pioneers shaping the future of AI.


As organizations advance toward production-level AI, establishing a robust evaluation framework is essential for long-term success.

 

The demand for more sophisticated AI solutions is growing, making Large Language Models (LLMs) an integral component of many AI strategies. They provide foundational capabilities that enhance various AI applications, driving efficiency, innovation, and improved decision-making.

 

However, LLMs introduce unique challenges challenges extending beyond POCs, including significant security, compliance, and regulatory risks. Effective LLM evaluations are essential to ensure AI systems function correctly, align with business goals, and remain compliant. 

At Thoughtworks, our world-class AI Research Lab — enhanced by our acquisition of Watchful — focuses on programmatic interpretability for LLMs, which is vital for ensuring that AI systems can be understood and trusted. 

 

Join us to delve into their innovative research and insights that will enhance your AI initiatives.

 

Melbourne

Tuesday, November 12, 2024 

Work Club Olderfleet

Level 35, 477 Collins Street, Melbourne VIC 3000

Google maps

 

Sydney

Tuesday, November 19, 2024 

Work Club Barangaroo

Suite 6.01, Level 6/201 Kent St, Sydney NSW 2000

Google maps

Marketo Form ID is invalid !!!

Thank you for your registration.

You will receive an email confirmation shortly.

Please note that this is an exclusive, invitation-only event with limited seating. Thoughtworks reserves the right to decline any registration. If you would like to us to extend an invitation to someone in your network, or have any other questions, please reach out to events-au@thoughtworks.com

"We are thrilled to be visiting Australia to engage with leaders and technologists shaping the future of AI. Our mission is to provide fast, transparent solutions that integrate domain expertise into models while clarifying their effectiveness."

- Shayan Mohanty and John Singleton, Thoughtworks AI Lab

 

 

Who should attend?

We encourage AI executive sponsors to attend alongside technical team leads to ensure alignment between strategic vision and technical implementation.

 

Executives: Attend the morning session to understand how evaluations drive success in GenAI. Learn effective strategies for assessing AI models and the key components of a solid evaluation framework that facilitate the shift from proof of concept to full-scale production.

 

Technologists: Stay until lunch for an in-depth look at LLM evaluations, including expanded definitions, current research, and future directions that can impact your strategies. Engage with peers over lunch to exchange insights and foster connections!

Agenda

8:30am to 9:30am

Registration and light refreshments

9:30am to 9:45am

Welcome

Andy Nolan, Director of Emerging Technologies, Thoughtworks SEAANZ

9:45am to 10:30am

Overview of AI evaluations

Shayan Mohanty and John Singleton representing Thoughtworks AI Lab


Executives and technologists are encouraged to attend this session, which will address the importance of evaluations for GenAI, including:

 

  • Effective strategies for guiding and evaluating AI models.

  • Key components of an effective evaluation framework.

  • The role of AI evaluations in facilitating the transition from proof of concept (POC) to full-scale AI production.

10:30am to 11:15am

Panel discussion: Leveraging AI evaluations for successful deployments

Executives and technologists are encouraged to attend this session. Our expert panel will explore:

 

  • The role high-quality evaluations play in meeting business goals and compliance.
  • Strategies for effective collaboration among teams to align evaluations with organizational objectives.
  • Key cost considerations when running evaluations.
  • How evaluations can enable effective feedback loops and reduce deployment risks.
  • Considerations for LLM security, compliance, and regulatory risks.

11:15am to 11:45am

Morning tea

11:45am to 12:45pm

Advanced evaluation and observability for LLMs

Shayan Mohanty, Head of AI Research, Thoughtworks


This session is more suited to those with a technical understanding of LLMs. Shayan will provide a deeper exploration of the latest findings and research direction of Thoughtworks' AI Lab, including:

 

  • Expanding the definition of evaluations.

  • Future work and potential impact.

     

This will be a highly interactive session, offering you the chance to ask challenging technical questions and delve into the complexities of LLM evaluation and observability.

12:45pm to 1:00pm

Conclusion

Andy Nolan, Director of Emerging Technologies, SEAANZ, Thoughtworks

1:00pm to 2:30pm

Lunch and networking

Speakers

Photo of Andy Nolan
Andy Nolan

Director of Emerging Technologies, Thoughtworks, SEAANZ

With over 20 years of experience, Andy specializes in applying emerging technologies like AI and computer vision to real-world challenges – developing innovative solutions for challenging environments and tightly regulated industries.

Photo of Shayan Mohanty
Shayan Mohanty

Head of AI Research, Thoughtworks AI Lab
Shayan's role is focused on bridging AI development and production. Previously CEO and Co-Founder of Watchful, he has led data engineering teams at Facebook and is a Guest Scientist at Los Alamos National Laboratory, with expertise in topics like Automata Theory and Machine Teaching.

Photo of John Singleton
John Singleton

Principal Program Manager, Thoughtworks AI Lab

John is a Principal Program Manager at Thoughtworks, bringing a wealth of experience from his previous role as Co-Founder and COO at Watchful which was recently acquired by Thoughtworks. He loves good food, playing disc golf, and foraging for mushrooms in his home of Sonoma, California.

Panelists

Photo of Mark Brown
Mark Brown

Head of Data and AI, ANZ, AWS (Melbourne event) 

Photo of Simon Johnston
Simon Johnston

Head of GTM Data and AI, ANZ, AWS (Sydney event) 

Photo of Lilly Ryan
Lilly Ryan

Global Secure Delivery Lead, Thoughtworks

Photo of Ned Letcher
Ned Letcher

Lead Data Science Engineer, Thoughtworks

Thoughtworks logo

Thoughtworks is a leading global technology consultancy that integrates strategy, design and software engineering to enable enterprises and technology disruptors across the globe to thrive as modern digital businesses.

 

Founded in 1993, we are over 10,500 Thoughtworkers strong across 48 offices in 19 countries. For 30+ years, we’ve delivered extraordinary impact together with our clients by helping them solve complex business problems with technology as the differentiator.