Activity: Bubble Popper {

Objectives

The idea

Using hand tracking we turn the user’s index finger into a pin on our program’s canvas. A bubble floats upward repeatedly and the user can pop the bubble with the pointy end of their pin-finger.

Create a new project

  1. Start a p5 template project in your activities folder and name it bubble-popper (template-p5-project.zip)
  2. Open the project folder in VS Code to start work
  3. Set the title of the project in index.html to something appropriate like Bubble Popper
  4. Commit the changes to your repository with a commit message

The plan

There are roughly four major things we need to do. We need to make hand tracking work, display a pin where the user’s index finger is, make a bubble move on the canvas, and finally make the pin pop the balloon.

So that’s the plan:

  1. Get Handpose working
  2. Draw a pin
  3. Draw and move a bubble
  4. Make the bubble pop

1. Get Handpose working

Technically the easiest way to get Handpose up and running with the webcam is probably just to take a pre-existing example and work from that, but let’s build it up from first principles instead.

Include ml5.js

First we clearly need to include the ml5.js library in our project

  1. Go to the ml5.js Homepage and navigate to “Getting Started” to find the script tag we need
  2. Include the script tag in index.html

Webcam

Let’s start the webcam…

  1. Create a canvas in setup() of your preferred dimensions (640x480?)
  2. Declare a global video variable
  3. In setup() use createCapture(VIDEO) to access the webcam and assign the return value to video
  4. Hide the video element so it doesn’t display on the page

If we run this program we should be asked for permission to use the webcam (if needed) and then should see the webcam light turn on.

Handpose

Let’s set up the Handpose model. We want to initialize the model and tell it we want to listen for “predict” events so we can track the user’s hand…

  1. Declare an undefined global handpose variable to store the model
  2. In setup() assign a new handpose model to handpose with three arguments
  1. Declare a global predictions variable and assign an empty array
  2. In setup() create a listener using .on() for the “predict” event with an anonymous callback function that

At this point if we run the program we should be able to go into the JavaScript console and, if we hold a hand in front of the webcam, we should see results coming in! Hand detection!

2. Draw a pin

Now that we have hand data, we can display our pin on the canvas. We’ll need to find the coordinates of the tip and base of the user’s index finger and draw the pin between them. To do this we need to remember/check how Handpose formats the prediction data. We can figure this out by reading the data and experimenting, or just remember it.

We’ll do all this in draw() since we’re displaying this on the canvas:

  1. Fill the background with a color (black?)
  2. Check if the predictions array has anything in it (is its length greater than 0?), if so:

If we run this program we should be able to hold up our hand and see a pin drawn wherever our index finger is!

(Note: if we need to debug one useful trick is to display the webcam data on the canvas so we can see what it’s seeing. Flip the image with ml5.flipImage() and display the result with image() so that it’s displayed correctly.)

3. Draw and move a bubble

To make having a pin for a finger more interesting we need something to pop. We’ll add a simple bubble (AKA a circle) that moves upward on the canvas and wraps around.

  1. Declare an undefined global bubble variable
  2. In setup() assign an object literal with properties for x, y, size, vx, vy
  1. In draw() after the predictions check, move and display the bubble using its properties

All going to the plan, when we run the program the bubble moves up the screen serenely and wraps when it hits the top. Right now our finger-pin does nothing because there’s no code for that yet…

4. Make the bubble pop

Now we want to make the bubble and the pin interact. In particular we want to check if the tip of the pin (the tip of the index finger) is inside the bubble’s area. If so the bubble should “pop” and reset to the bottom of the canvas.

We’ll do this in draw() inside the predictions check, after we’ve calculated the positions of the index finger:

  1. Calculate the distance from the tip of the index finger to the center of the bubble
  2. Write an if statement that checks if this distance means the tip of the pin is inside the bubble (based on the bubble’s size)

Now when we run the program we should be able to pop the bubble with the pin! We are literally moving our real hand around and in the program it’s represented as a pin that can pop a bubble. Ridiculous!

5. Improve the program

As always, there are improvements that can be made. Consider

Done

As with any simple program, we can imagine all kinds of improvements and changes?

}