CamoBot

IoT robot camera car that helps film production by following actors via computer vision or director's gesture control

Comment 1

Camobot

Inspiration

Known as the entertainment capital of the world, Los Angeles is the city for television and filmmaking. During the process of filmmaking, there is often the need to track actors /actresses and take the close-up, and the current process needs one cameraman and its assistant to do the job. One scene might need to shoot multiple times to get different kinds of angles. Also, there are scenes involving explosives and might be dangerous.

In order to improve the efficiency and safety, Camobot is an IoT camera robot car designed to replace the assistant and track the actors /actresses automatically using Computer Vision as well as giving directors the option to control with gestures.

What it does

Camobot is a IoT robot camera car that aims that helps film production by tracking actors automatically via computer vision or following the directors’ gesture control.

How we built it

We used Google Compute Engine on Google Cloud Platform as the core platform to build our server that connects to multiple device and cloud service. Google IoT Service is a very important component in our system. The communication between Raspherry Pi on the robot and main server is established through Google IoT Core via MQTT bridge. The server subscribes to IoT service to receive data from the hardware and also sends commands to the robot when a gesture event is sent from Leap Motion. Meanwhile, the server also provides http APIs for Leap Motion to send gesture data so that the server can send corresponding moving command to the robot, allowing the user to control the robot remotely by gestures. To further improve the intelligence of the remote, we connect our server with Google Cloud Vision API. By analyzing the images sent from the robot in real time with face recognition API, we also applied our own mathematical methods to analyze the images and generates appropriate commands to the robot so that the robot can automatically adjust its position to keep a moderate distance and degree to film the actor/actress when he/she moves around the camera.

Challenges we ran into

  1. Cannot remotely control the movement of Camobot through Rasberry-pi
  2. Google IoT pub/sub message receives past messages and cannot receive real-time message
  3. Leap Motion detection latency

Accomplishments that We're proud of

We finished everything we set up to do!

  1. Camobot can move according to the gesture detected by Leap Motion
  2. Camobot can detect human faces and follow the movement of the face
  3. IoT framework set up successfully

What we learned

  1. Google Cloud Platform
  2. Program on Raspberry-pi
  3. Get the detection of gesture from leap motion

What's next for Camobot

Multiple Camobots in one setting

In the film production, one scene usually needs various angels. Multiple Camobots can be deployed in one setting and works together to shoot from multiple angles.

Subject of filming

Camobot can be extended to detect other objects with proper usage of image detection API on different objects. For example, Camobot can be used to film endangered animals for documentary purposes.

Latency

We hope we can further improve the latency issue for the whole process.

Share this project:

Updates