Difference between revisions of "Gesture-Controlled Tello Drone- Rapolas Kairys"
(→Gesture Controlled Tello Drone Project) |
(→Gesture Controlled Tello Drone Project) |
||
| Line 5: | Line 5: | ||
* Explore computer vision and machine learning for gesture recognition. | * Explore computer vision and machine learning for gesture recognition. | ||
* Demonstrate real-time controls using multithreading to have a smooth video feed. | * Demonstrate real-time controls using multithreading to have a smooth video feed. | ||
| + | |||
=== Description === | === Description === | ||
| Line 14: | Line 15: | ||
* Custom gesture logic to determine gestures (LEFT arm up, RIGHT arm up, both arms up, or none). | * Custom gesture logic to determine gestures (LEFT arm up, RIGHT arm up, both arms up, or none). | ||
| + | === Controlls === | ||
When the user performs a gesture in front of the laptop camera, the system detects it and translates it into a drone command: | When the user performs a gesture in front of the laptop camera, the system detects it and translates it into a drone command: | ||
| Line 23: | Line 25: | ||
* The command has to be held for 1.5 seconds to take effect. This helps prevent accidental commands and ensure safety. | * The command has to be held for 1.5 seconds to take effect. This helps prevent accidental commands and ensure safety. | ||
* While the program is running and a drone is connected pressing "t" makes the drone take off/land. Pressing "q" shuts down the program. | * While the program is running and a drone is connected pressing "t" makes the drone take off/land. Pressing "q" shuts down the program. | ||
| + | |||
| + | |||
| + | === Steps Taken === | ||
| + | |||
| + | # Environment Setup Installed necessary Python libraries: mediapipe, opencv-python, djitellopy. Created a virtual environment to keep dependencies organized. | ||
| + | |||
| + | # Implemented Gesture Detection | ||
| + | |||
| + | Used Mediapipe’s Pose Estimation to identify shoulder and elbow landmarks. | ||
| + | Determined a gesture by comparing y-coordinates of elbows vs. shoulders (e.g., elbow above the shoulder). | ||
| + | Drone Controller | ||
| + | |||
| + | Developed a background thread to send blocking commands (like takeoff()) so the main loop remains responsive. | ||
| + | Stored only one command at a time (no queue) and executed it before accepting the next. | ||
| + | Main Application Loop | ||
| + | |||
| + | Captured frames from the laptop camera in a separate thread to avoid UI lag. | ||
| + | Performed gesture recognition in the main loop, and if needed, updated the drone command. | ||
| + | Displayed relevant HUD info: active command, battery status, etc. | ||
| + | Testing and Tuning | ||
| + | |||
| + | Adjusted arm-raise thresholds for better recognition accuracy. | ||
| + | Verified drone responsiveness and safe takeoff/landing. | ||
Revision as of 16:11, 2 February 2025
Contents
Gesture Controlled Tello Drone Project
Goal of the Project
The main objective of this project is to control a Tello drone using gesture recognition from a laptop camera. Users can raise their arms in specific poses to make the drone move left, right, up or toggle flight (take off/land) by holding an “UP” pose for four seconds. This project aims to:
- Provide a way to interact with and control a small drone using hand gestures.
- Explore computer vision and machine learning for gesture recognition.
- Demonstrate real-time controls using multithreading to have a smooth video feed.
Description
The system uses:
- MediaPipe Pose Estimation to detect body landmarks (shoulders, elbows).
- OpenCV for video processing.
- DJITelloPy library to send commands to the Tello drone over Wi-Fi, handling takeoff, landing, and movement.
- Multithreading to ensure that drone commands (which can block) do not freeze the camera feed or the user interface.
- Custom gesture logic to determine gestures (LEFT arm up, RIGHT arm up, both arms up, or none).
Controlls
When the user performs a gesture in front of the laptop camera, the system detects it and translates it into a drone command:
- Holding both arms raised for 4 seconds toggles flight (either takeoff or land).
- Once in the air, raising both arms makes the drone go up.
- Raising only the left or right arm makes the drone move left or right.
- Doing neither results in a hover command.
- The drone moves 30cm for all commands, which can be changed in drone_controller.py.
- The command has to be held for 1.5 seconds to take effect. This helps prevent accidental commands and ensure safety.
- While the program is running and a drone is connected pressing "t" makes the drone take off/land. Pressing "q" shuts down the program.
Steps Taken
- Environment Setup Installed necessary Python libraries: mediapipe, opencv-python, djitellopy. Created a virtual environment to keep dependencies organized.
- Implemented Gesture Detection
Used Mediapipe’s Pose Estimation to identify shoulder and elbow landmarks. Determined a gesture by comparing y-coordinates of elbows vs. shoulders (e.g., elbow above the shoulder). Drone Controller
Developed a background thread to send blocking commands (like takeoff()) so the main loop remains responsive. Stored only one command at a time (no queue) and executed it before accepting the next. Main Application Loop
Captured frames from the laptop camera in a separate thread to avoid UI lag. Performed gesture recognition in the main loop, and if needed, updated the drone command. Displayed relevant HUD info: active command, battery status, etc. Testing and Tuning
Adjusted arm-raise thresholds for better recognition accuracy. Verified drone responsiveness and safe takeoff/landing.
