ML5 Key Points Tool

A new feature for my ‘p5 web editor debugger’ project that will facilitate users with using the ML5 library in their sketches on the p5 web editor.

https://docs.google.com/presentation/d/1rp56P-qXJlDfpzbr5lr3H5IR4kbmGs1YKaYUvGqrN0I/edit?usp=sharing

Github Repository:

GitHub - ch3926/p5-web-debugger at extension

Abstract

My final project aims to build upon the ml5 library and make starting out with machine learning even more approachable for beginners. This final project aims to prototype a ml5 Keypoints tool that allows users to select points on image representations of ml5 models (poseNet, Handpose, FaceMesh etc). Then they can choose what functionality to apply to these points (find distance between two pts, replace ‘cursor’ with this point, etc). I am developing a ‘Debugger’/ all around toolset for adding functionality to the p5 web editor. Currently functionality includes: Debugging (play, pause, stepping through frames), and a chatGPT personal assistant. My hope is to eventually add the ml5 Keypoints tool to this set. However, for the scope of this final project, I aim to create a proof-of-conscept set of example sketches.

Inspiration

I was greatly inspired by the progress on the p5 Debugger. I saw the potential of this tool expanding beyond a basic debugger and becoming an all around tool to help beginners with both p5 and ml5.js. When I though about what type of ml5 functionality I could feasibly add to the debugger, the idea was pretty much instant. During this semester, our Machine Learning For the Arts class used ml5 a lot, and I noticed that one of the most tedious and annoying things when dealing with any of the pose models was actually simply being able to find the right selectors. Once you had the selectors, you could easily, for example, use your index finger as a mouse. Additionally, at the start of this semester, I used Figma to mock up a configurator website for the VIP program’s (Project Mjolnir) bike. And this got me thinking about user friendly design and helped me make the connection between the Debugger tool and how to make ml5 more approachable to new users! Additionally, I had the opportunity to work on the ml5-next-gen library this summer 2023. This experience prompted me to feel very passionate about helping further ml5.js’ goals of more accessible machine learning! All of these reasons pointed me towards creating the ml5 KeyPoints tool!

Audience

My project aims to extend the goals of both p5.js and ml5.js. I hope to make programming and specifically machine learning more accessible to artists, beginners, and anyone who feels intimidated by using ml5 but are interested in giving it a try. This tool also has a focus on making teaching with ml5 easier for instructors. I also hope that by extending functionality on a free/open-source ecosystem, I can inspire others to contribute more to open sourced projects.

Challenges

• Tool Format: I struggled a lot with trying to set the scope for this project. I am really excited to add the Keypoints tool to my Debugger tool. However, this might be too ambitious of a goal for the remaining 3 weeks of this semester. Upon talking to my professor, I was advised to focus on creating a proof of conscept / prototype of my idea. This will help quickly highlight the significance of my idea without getting bogged down by the difficulties of creating a Chrome extension (which I am learning for the first time). Therefore, I have settled on the mini-goal of creating a series of example sketches to illustrate the pieces of my tool. In the future, I hope to add it to the Chrome extension.
• User Experience: How do I optimize for beginners and education? How do I make my product intuitive. How can I predict what issues beginners might face when starting out with ml5.js? I hope to do lots of user testing throughout my work so I can continuously learn and improve my project design.
• ml5-next-gen: I will be working with ml5-next-gen which has less resources and examples. Many of the models have been updated between the original and new release of ml5.js. While much remains similar, there are differences in how functions are initialized, added parameters etc, that will require new example codes to be made. I hope to contribute to these new example codes and expand the resources out there for people starting with or migrating to ml5-next-gen!

Visual Reference