Leap Motion gesture controlled Robot | Leap Robotic Arm Processing Arduino

Leap Gesture controlled Robot

In this project we are going to control Robot with hand gestures using Leap-motion. There are no sensors and actuators involved or placed over the hands.

Its a natural-user interface and having plug and play facilities. We are controlling Robotic hand gripper with the help of Arduino and processing software. In this tutorial we will discuss about gesture robot and Arduino robot arm code

Contents for Leap Motion Gesture Robot:

1. What is Leap Motion controller gestures detector leap sensor
2. Introduction of software Arduino + Processing
3. How to control Leap Motion sensor with Arduino
4. How to control gesture controlled Robot
5. How to Code for Leap Motion.

01: What is Leap Motion Controller gestures detector sensor

Leap motion 3D hand gesture detector sensor changed the working vision. A non-contact sensor tracks Five fingers, Hands, Arm.
The functionality of Motion sensor to detect the position of Hands, the Wrist, Palms and also X, Y, Z orientations of hands. Its highly effective accuracy provides many software’s to interface with it. Leap Motion controller consist of two multi-tracking cameras that takes 200 frames per second. Its three Infra Rays led gives a wide range of 1*1 feet tracking area. The minimum distance for tracking hands is 1 inch.

The main purpose of Leap Motion is to give way of controlling real world hardware with 3D technology controls. Until yet people play games with gestures in computers. Now time to change and come up in real world hardware. Motion sensor turns the intentions of projects like if you remember that when the touch screen mobiles are introduced no one is interested in phone with buttons and that established a great change in life style. Think about computers.

We used keyboard and mouse with it. If we can able to work without keyboards and mouse that’s another impressive part. This 3D technology gives us way to control computers with gestures. Detecting a human hand is very difficult because orientations, fingers, position of hands. Knowing this device its easier more stable more efficient usable without delays have high accuracy.

02: Introduction of software’s Arduino + Processing

Arduino is open source software and hardware which is extremely used in projects. It has simplest programming and programming software called Arduino IDE. Hardware side of Arduino is consist of many types of Arduino boards like UNO, NANO, MEGA etc. It has many pins for controlling purpose depending on board type you are using. Its has analog pins, digital pins for inputs and outputs.

Easy to use with Leap Motion technology even don’t have to add any additional library in it for motion detection integration. further we will discuss the details of coding with motion detector sensor, gesture robot and Arduino robot arm code. For more detail about Arduino software and hardware boards

Read : getting-started-with-Arduino-microcontroller

Processing is a open source Java based programming language and a flexible software IDE integrated development environment made for controlling electronics, visual arts, image processing, gaming and technology. Like Arduino software the main idea of processing is to give a platform for non programmers to start computer programming without any advance knowledge of programming.

It is simplified computer user interface graphics software. Processing software programming is also called sketch as its software gives variety of libraries and classes. You can also make your own classes and it has very basic programming so that students of any age easily manipulate it for graphics, image-processing, Leap Motion controller etc. It encourage and support engineers, programmers to design creative ideas. for more detail Read:

Processing software for leap motion

03: How to control Leap Motion Controller with Arduino

We cannot directly controlled Leap Motion with Arduino we must need another software to receive data from motion detector sensor. Arduino will help to convert gestures recognizing data into the hardware electronics controlling purpose. We used Processing software to recognize and tracks the gestures of hands, fingers, arm, wrist positions in coordinates system and send data to Arduino.

Arduino used the data of gestures from processing and process it to move motors. It have built-in libraries to communicate with processing so we do not install any extra library in it. On the other hand processing software needs to add extra library to communicate with gestures recognition technology.

Adding library in Processing software.

• Go to Sketch.

• Click Import library

• Add library

Search screen open then type ” Leap Motion for Processing ” and add library. This library is completely fully developed for taking inputs from motion detector device. If you check its examples this library have three examples for conform working of  motion detecting device that detects and shows your both hands, fingers, id, hand strength, palm position, hand name either its right hand or left hand and also shows time duration.

I am giving you an example just copy and paste it on processing and enjoy the working of gestures recognizing technology. Remember first you have to install library and plugin Leap Motion controller device then its example working.

import de.voidplus.leapmotion.*;

// ======================================================
// Table of Contents:
// ├─ 1. Swipe Gesture
// ├─ 2. Circle Gesture
// ├─ 3. Screen Tap Gesture
// └─ 4. Key Tap Gesture
// ======================================================

LeapMotion leap;

void setup(){
size(800, 500);
background(255);
// ...

leap = new LeapMotion(this).allowGestures();  // All gestures
// leap = new LeapMotion(this).allowGestures("circle, swipe, screen_tap, key_tap");
// leap = new LeapMotion(this).allowGestures("swipe");  // Leap detects only swipe gestures
}

void draw(){
background(255);
// ...
}

// ======================================================
// 1. Swipe Gesture

void leapOnSwipeGesture(SwipeGesture g, int state){
int     id               = g.getId();
Finger  finger           = g.getFinger();
PVector position         = g.getPosition();
PVector positionStart    = g.getStartPosition();
PVector direction        = g.getDirection();
float   speed            = g.getSpeed();
long    duration         = g.getDuration();
float   durationSeconds  = g.getDurationInSeconds();

switch(state){
case 1: // Start
break;
case 2: // Update
break;
case 3: // Stop
println("SwipeGesture: " + id);
break;
}
}

// ======================================================
// 2. Circle Gesture

void leapOnCircleGesture(CircleGesture g, int state){
int     id               = g.getId();
Finger  finger           = g.getFinger();
PVector positionCenter   = g.getCenter();
float   radius           = g.getRadius();
float   progress         = g.getProgress();
long    duration         = g.getDuration();
float   durationSeconds  = g.getDurationInSeconds();
int     direction        = g.getDirection();

switch(state){
case 1: // Start
break;
case 2: // Update
break;
case 3: // Stop
println("CircleGesture: " + id);
break;
}

switch(direction){
case 0: // Anticlockwise/Left gesture
break;
case 1: // Clockwise/Right gesture
break;
}
}

// ======================================================
// 3. Screen Tap Gesture

void leapOnScreenTapGesture(ScreenTapGesture g){
int     id               = g.getId();
Finger  finger           = g.getFinger();
PVector position         = g.getPosition();
PVector direction        = g.getDirection();
long    duration         = g.getDuration();
float   durationSeconds  = g.getDurationInSeconds();

println("ScreenTapGesture: " + id);
}

// ======================================================
// 4. Key Tap Gesture

void leapOnKeyTapGesture(KeyTapGesture g){
int     id               = g.getId();
Finger  finger           = g.getFinger();
PVector position         = g.getPosition();
PVector direction        = g.getDirection();
long    duration         = g.getDuration();
float   durationSeconds  = g.getDurationInSeconds();

println("KeyTapGesture: " + id);
}

04: How to control Robotic hand gripper

We are using Servo motors for this project. The Servo motor is a rotatory actuator that controls angular movement. Servo motor works between 0 to 180 degree. For gripper we are using 0 to 180 degree. When the gripper is open it means servo is rotates to 180 degree and when gripper is closed it means servo is at 0 degree or vice versa.

Here we discuss the servo controlling and connections with Arduino. Servo motors have three wires Red, Black, White. The Red wire attached with Arduino 5v  while the Black one is ground and the White is on pin number which is used to control servo angle in programming. We are using pin number 3 in Arduino microcontroller.

05: How to Code for Leap Motion Controller

First command tells the Leap motion availability and library. The second one is to serial communication. Now we mention the fingers to control grab strength. By grab strength we give the commands to open and close the paw. In example above you will see the grab strength is between 0 to 1 in float values. Fingers are identified by these commands.

PVector Thumb=null,Index=null,Middle=null;

This command describe the thumb, index, and middle finger. PVector describes the position vectors. For serial monitor with Arduino microcontroller must tell the port number in which your Arduino microcontroller is attached with your pc, laptop etc. also mention the baud rate speed for transfer data from processing. Here we use

You can change baud rate if needed. Change the “com0” with com you are connected your microcontroller. Now time to place positions for fingures. By taking these positions it begins to differentiate the grab strength.

Getting index and middle finger position is same like getting thumb. As mention above strength of hand is from 0 to 1 these values used in map command to integrate all values between 0 to 1 into servo motor angles 0 to 180 degree. This command saved in some variable then send to Arduino software then the actual working of Arduino software begins.

In Arduino side we define the servo motor pin number in which servo is attached. Read data via serially from processing and write them to hardware. If serial available read the values coming from leap motion controller via processing for hand gripper and write it on servo motor.

Now servo motor moves with closing and opening the hands. It detects the values between 0 to 1 in float type and the map command make it divide in small steps for servo 180 degree angles and write it to servo.

myservo1.attach(3);

while(Serial.available())

Serial.read();

myservo1.write;

For code in align form see: servo-motor-using-leapmotion-Arduino

Hope you find it knowledgeable and learn a lot with this gesture controlled robot post. Keep in touch with us. There are other Tutorials available on Leap Motion sensor and gesture controlled robot .If you have any query we are always be here for your help. Stay connected with us. There will be much more tutorials and lessons regarding Leap motion technology. Thanks.