One of the visions in AI based robotics are household robots that can autonomously handle a variety of meal preparation tasks. Based on this scenario, we present a best practice tutorial on how to create actionable knowledge graphs that a robot can use for execution of task variations of cutting actions. We implemented a solution for this task that integrates all necessary software components in the framework of the robot control process. In the context of this tutorial, we focus on knowledge acquisition, knowledge representation and reasoning, and simulating robot action execution, bringing these components together into a learning environment that – in the extended version – introduces the whole control process of Cognitive Robotics. In particular, the Tutorial will detail necessary concepts a knowledge graph should include for robot action execution, how web knowledge can be automatically acquired for the domain of cutting fruits, and how the created knowledge graph can be used to let robots execute tasks like slicing a cucumber or quartering an apple. The learning environment follows an immersive approach, using a physics-based simulation environment for visualization purposes that helps to illustrate the concepts taught in the tutorial.
Using Jupyter Notebooks in a Docker environment, our learning environment is easily accessible without having to install different software packages and is independent of the learners’ technical setup.
The tutorial is a half-day event focused on two, consecutive hands-on sessions,. The general application domain, in which the hands-on sessions are embedded, are meal preparation tasks. For the sessions, the focus lies on the task of “Cutting fruits & vegetables“.
The first hands-on session is focused on the extraction of relevant action and object knowledge from the Web, whereas the other uses the collected knowledge for the parameterisation of generalised manipulation plans using the collected knowledge through knowledge graphs.
Program @ EKAW 2024 (26th November 2024): tba
More information on the topics presented in the tutorial can be found on our website.
The material for the tutorial is available on GitHub.