profile
viewpoint

Ask questionsGet hand_tracking_cpu source code to extract detected hand metadata

By running this code below, I am able to get the demo app to run but, where can I get the source code so that I can get the hand metadata regarding finger positions and joints?

GLOG_logtostderr=1 bazel-bin/mediapipe/examples/desktop/hand_tracking/hand_tracking_cpu \
    --calculator_graph_config_file=mediapipe/graphs/hand_tracking/hand_tracking_desktop_live.pbtxt

I will like to modify: bazel-bin/mediapipe/examples/desktop/hand_tracking/hand_tracking_cpu so I can output a file with hand metadata for sign language analysis. But this seems to be a binary file. How can I use this or integrate this with a new app?

google/mediapipe

Answer questions research506

@research506 Please look at https://mediapipe.readthedocs.io/en/latest/multi_hand_tracking_desktop.html for how to build and run multi-hand tracking example on desktop.

On desktop, you can add an output stream poller to demo_run_graph_main.cc or demo_run_graph_main_gpu.cc, such as

ASSIGN_OR_RETURN(mediapipe::OutputStreamPoller multi_hand_landmarks_poller,
                   graph.AddOutputStreamPoller("multi_hand_landmarks"));

to listen for output packets on a particular output stream, such as "multi_hand_landamrks" above. See https://mediapipe.readthedocs.io/en/latest/hello_world_desktop.html for example.

Then, on every frame, check if a packet is available in the output stream:

mediapipe::Packet multi_hand_landmarks_packet;
if (!multi_hand_landmarks_poller.Next(&multi_hand_landmarks_packet)) break;
const auto& multi_hand_landmarks = multi_hand_landmarks_packet.Get<std::vector<std::vector<mediapipe::NormalizedLandmark>>>();

Note that you need a dependency on

#include "mediapipe/framework/formats/landmark.pb.h"

in the file you are modifying (and modify the corresponding rule in BUILD file as well).

And then you should be able to do what you want with these landmarks, such as print them:

int hand_index = 0;
for (const auto& hand_landmarks : multi_hand_landmarks) {
  int landmark_index = 0;
  for (const auto& landmark : hand_landmark) {
    std::cout << "[Hand<" << hand_index << ">] Landmark<" << landmark_index++ << ">: (" << landmark.x() << ", " << landmark.y() << ", " << landmark.z() << ")\n";
  }
 std::cout << "\n";
 ++hand_index;
}

My brother, this was the droid I was looking for. I will update this thread with a link to my repo where the code can be found. I owe you a beer

useful!

Related questions

SSL certification error while downloading dependency libraries via Bazel hot 1
Build of Hello World Failed on Darwin (MacOS) hot 1
Build of Hello World Failed on Darwin (MacOS) hot 1
Bazel build failed in Android Studio hot 1
Fail to pull in MediaPipe's external dependencies due to unstable internet connection hot 1
at com.google.mediapipe.apps.objectdetectioncpu.MainActivity.<clinit>(MainActivity.java:50) hot 1
Hand Tracking Desktop GPU Build Error hot 1
ERROR: An error occurred during the fetch of repository 'bazel_skylib' hot 1
Bazel build failed to fetch Maven dependency hot 1
Running error hot 1
Hello World failed build on Raspberry Pi - mediapipe hot 1
Accessing landmarks, tracking multiple hands, and enabling depth on desktop hot 1
Unable to load the hand detection model hot 1
build aar failed hot 1
Error in building TensorFlow Object Detection Demo hot 1
Github User Rank List