Large Language Models (LLMs) can help add many new and exciting functionality to embedded devices. Like all machine learning, this can be resource intensive on constrained embedded devices. This session looks at ways to meet machine learning's resource requirements from the prospective of an embedded developer.
This session will start by looking at potential embedded applications along with limitations that needs to be considered. A quick overview of machine learning will be provided to help enable fellow embedded developers to understand machine learning terminologies. Non LLM machining learning alternatives with similar functionality will be looked at.
Focus will be on local execution/inference using openly available models but cloud options will be discussed. Hardware and software demands will be looked at along with possible trade offs to balance functionality vs cost. Measurements data with open models on common current embedded Linux platforms will be used to illustrate some of the trade offs and drawbacks. Most of the session will be on the device/inference end but backend/training will be touched upon.
