Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

add spot_human_recognition demo_code #1387

Open
wants to merge 23 commits into
base: master
Choose a base branch
from

Conversation

heissereal
Copy link

spotの360度カメラで、人物を認識したら、そこに近づいてふりむいて声をかけ、「あっちに行きたい」と指を指した方向へと先導したいです。 @mqcmd196 cc. @sktometometo @tkmtnt7000

@mqcmd196
Copy link
Member

@heissereal
PRでディレクトリ構成を変更しました.

source ~/spot_ws/devel/setup.bash
roscd jsk_2023_04_human_approach

とすると,作業ディレクトリに移って作業できます.

今までやってくれた作業はあるんだけど,今回BoundingBoxが360度カメラの画像から出せるlaunchがあることがわかったので,
/rect_array_in_panorama_to_boundig_box_array/bbox_array をSubscribeして,そこにgo-posする処理を書いてみましょう.
ヒント: 型は jsk_recognition_msgs/BoundingBoxArray,で位置情報とかが入っています.

手を検出するのは明日やってみます

@mqcmd196
Copy link
Member

mqcmd196 commented Apr 17, 2023

/skeleton_with_depth/output/skeleton (型はjsk_recognition_msgs/HumanSkeletonArray) に奇跡的にこの中身のtopicが入ることがある.

---
header: 
  seq: 4862
  stamp: 
    secs: 1681715628
    nsecs: 217715354
  frame_id: "hand_color_image_sensor"
human_ids: []
skeletons: 
  - 
    header: 
      seq: 4862
      stamp: 
        secs: 1681715628
        nsecs: 217715354
      frame_id: "hand_color_image_sensor"
    bone_names: 
      - right shoulder->right elbow
    bones: 
      - 
        start_point: 
          x: 2.3581541222927527
          y: 0.3860775946041869
          z: 1.5579999685287476
        end_point: 
          x: 1.8728612915368605
          y: 0.7816122661615572
          z: 1.5609999895095825
  - 
    header: 
      seq: 4862
      stamp: 
        secs: 1681715628
        nsecs: 217715354
      frame_id: "hand_color_image_sensor"
    bone_names: []
    bones: []

重要なのはbone_namesのフィールドに,elbowが含まれていること.このときのbonesのフィールドの start_point end_point の2点を拾ってこれれば,腕の指す方向ベクトルがわかる.

@mqcmd196
Copy link
Member

今まで僕がSpot体内に入って,認識器のlaunchを上げていたのですが,それをやってもらいます

ssh miyamichi@belka

でSpot体内に入ります.でSpot体内で

tmux

と打って,tmuxを立ち上げます.その中で

source ~/ros/spot_ws/devel/setup.bash
roslaunch jsk_2023_04_human_approach recognition.launch

を立ち上げます.そうすると,360度カメラが見えるようになり,360度カメラの人認識,ハンドカメラによる人の姿勢認識が走ります.
このtmux はターミナルの中でターミナルを立ち上げるツールみたいなもので,手元のsshが切れたとしても体内のターミナルに残り続けます.それに接続するには

tmux a

を打ちます.

@mqcmd196
Copy link
Member

体内で上げるlaunch (jsk_2023_04_human_approach/launch/recognition.launch) に

360度カメラでの人の位置認識

<include if="$(arg coral)" file="$(find coral_usb)/launch/edgetpu_panorama_object_detector.launch">
  <arg name="INPUT_IMAGE" value="/dual_fisheye_to_panorama/output"/>
</include>

ハンドカメラでの人の姿勢認識

<include if="$(arg coral)" file="$(find coral_usb)/launch/edgetpu_human_pose_estimator.launch">
    <arg name="INPUT_IMAGE" value="/spot/camera/hand_color/image"/>
</include>

があって,どっちかコメントインしたらどっちかコメントアウトしないと動かないです.これも直しておきます

@heissereal
Copy link
Author

transformのところでSegmentation Faultが起こります…
Screenshot from 2023-04-21 10-54-41

@mqcmd196
Copy link
Member

@YUKINA-3252 @tkmtnt7000 @Kanazawanaoaki Could you solve this error?

@heissereal
Copy link
Author

以下のように表示されて、recognition.launchが立ち上がっていないようですが、どこを直したらいいでしょうか

process[insta360/insta_camera_node-1]: started with pid [7198]
process[split_image-2]: started with pid [7203]
process[dual_fisheye_to_panorama-3]: started with pid [7209]
process[camera_tf_publisher-4]: started with pid [7212]
type is jsk_perception/DualFisheyeToPanorama
process[rect_array_in_panorama_to_bounding_box_array-5]: started with pid [7230]
process[miyamichi_demo_nodelet_manager-6]: started with pid [7234]
process[nodelet-7]: started with pid [7240]
process[coral_input_image_mux-8]: started with pid [7257]
[ INFO] [1682073108.894982065]: Loading nodelet /nodelet of type depth_image_proc/register to manager miyamichi_demo_nodelet_manager with the following remappings:
[ INFO] [1682073108.907839539]: /depth/camera_info -> /spot/camera/hand_depth/camera_info
[ INFO] [1682073108.908416156]: /depth/image_rect -> /spot/camera/hand_depth/image
[ INFO] [1682073108.908768035]: /depth_registered/image_rect -> /spot/camera/hand_depth/depth_registered/image
[ INFO] [1682073108.908928155]: /rgb/camera_info -> /spot/camera/hand_color/camera_info
[ INFO] [1682073108.930265040]: waitForService: Service [/miyamichi_demo_nodelet_manager/load_nodelet] has not been advertised, waiting...
[ INFO] [1682073108.956762570]: Initializing nodelet with 8 worker threads.
process[edgetpu_node_manager-9]: started with pid [7353]
[ INFO] [1682073108.982650093]: waitForService: Service [/miyamichi_demo_nodelet_manager/load_nodelet] is now available.
process[skeleton_with_depth-10]: started with pid [7372]
[ INFO] [1682073109.321729514]: advertising
[ INFO] [1682073109.357527520]: Opening camera with vendor=0x2e1a, product=0x1000, serial="", index=0
unsupported descriptor subtype VS_STILL_IMAGE_FRAME
unsupported descriptor subtype VS_COLORFORMAT
unsupported descriptor subtype VS_COLORFORMAT
attempt to claim already-claimed interface 1
[ WARN] [1682073109.515370035]: Unable to set scanning_mode to 1
[ WARN] [1682073109.515485382]: Unable to set auto_exposure to 2
[ WARN] [1682073109.515679040]: Unable to set auto_exposure_priority to 1
[ WARN] [1682073109.515731994]: Unable to set exposure_absolute to 1
[ WARN] [1682073109.515907667]: Unable to set auto_focus to 1
[ WARN] [1682073109.515963428]: Unable to set focus_absolute to 22074
[ WARN] [1682073109.516130940]: Unable to set iris_absolute to 0
[ WARN] [1682073109.516310468]: Unable to set pantilt to 0, 0
[ INFO] [1682073109.625518942]: using default calibration URL
[ INFO] [1682073109.633654133]: camera calibration URL: file:///home/miyamichi/.ros/camera_info/camera.yaml
[ INFO] [1682073109.633782795]: Unable to open camera calibration file [/home/miyamichi/.ros/camera_info/camera.yaml]
[ WARN] [1682073109.633820344]: Camera calibration file /home/miyamichi/.ros/camera_info/camera.yaml not found.
[ INFO] [1682073109.655388587]: light_compen : false
[ INFO] [1682073109.658198510]: refine_align : false
[ INFO] [1682073109.658240188]: fovd         : 210.000
[ INFO] [1682073109.658255857]: save_unwarped: false
[ INFO] [1682073109.658268669]: mls_map_path : /home/miyamichi/ros/spot_ws/src/jsk_recognition/jsk_perception/config/dual_fisheye_to_panorama/fisheye_stitcher_grid_xd_yd_2000x1000.yml.gz
[ INFO] [1682073109.658284061]: blend_image_height : 1000
[ INFO] [1682073109.658297324]: blend_image_width  : 2000
[ INFO] [1682073109.658309911]: blend_param_p_wid : 30
[ INFO] [1682073109.658326595]: blend_param_p_x1 : 75
[ INFO] [1682073109.658339847]: blend_param_p_x2 : 885
[ INFO] [1682073109.658351456]: blend_param_row_start : 295
[ INFO] [1682073109.658363641]: blend_param_row_end : 660
[ INFO] [1682073109.658374951]: output_image_height : 1000
[ INFO] [1682073109.658384406]: output_image_width  : 2000
[INFO] [1682073110.257067]: starting edgetpu_panorama_object_detector
[INFO] [1682073110.266629]: Using transport raw
[ INFO] [1682073110.305020808]: initialize stitcher w:2000 h:1000
Initializing necessary parameters..
[WARN] [1682073114.430297]: subscribing topic seems not to be pulished. waiting... Error: timeout exceeded while waiting for message on topic ~input_class
[WARN] [1682073114.793678]: [/split_image] subscribes topics only with child subscribers. Set '~always_subscribe' as True to have it subscribe always.
[WARN] [1682073115.021633]: [/skeleton_with_depth] subscribes topics only with child subscribers. Set '~always_subscribe' as True to have it subscribe always.
[ INFO] [1682073122.394820020]: Bond broken, exiting
[nodelet-7] process has finished cleanly
log file: /home/miyamichi/.ros/log/94c61ef0-de72-11ed-9c6f-a8a1590e9101/nodelet-7*.log
^C[skeleton_with_depth-10] killing on exit
[edgetpu_node_manager-9] killing on exit
[coral_input_image_mux-8] killing on exit
[miyamichi_demo_nodelet_manager-6] killing on exit
[rect_array_in_panorama_to_bounding_box_array-5] killing on exit
[camera_tf_publisher-4] killing on exit
[dual_fisheye_to_panorama-3] killing on exit
[insta360/insta_camera_node-1] killing on exit
[split_image-2] killing on exit
[WARN] [1682073130.376487]: subscribing topic seems not to be pulished. waiting... Error: rospy shutdown
[INFO] [1682073130.524221]: Node is successfully initialized
shutting down processing monitor...
... shutting down processing monitor complete
done


@heissereal
Copy link
Author

上のエラーは、Spotを再起動したら直りました。
cc: @tkmtnt7000

@k-okada
Copy link
Member

k-okada commented Apr 21, 2023

治ったようで良かったです.
メール書きかけていた,より詳しい情報を知る方法,のメモをおくります.
===
これは,

[WARN] [1682073115.021633]: [/skeleton_with_depth] subscribes topics only with child subscribers. Set '~always_subscribe' as True to have it subscribe always.
[ INFO] [1682073122.394820020]: Bond broken, exiting
[nodelet-7] process has finished cleanly
log file: /home/miyamichi/.ros/log/94c61ef0-de72-11ed-9c6f-a8a1590e9101/nodelet-7*.log <<立ち上がってもここで止まってしまう
^C[skeleton_with_depth-10] killing on exit << ここは自分でC-cした

という理解でいいかな.まだ解決していなければ,解決方法ではないけど,より詳しい状況確認法は以下かな.

  1. ROSCONSOLE_FORMAT=[${severity}] [${time}]: [${node}] ${message} roslaunch --screen jsk_2023_04_human_approach recognition.launch としてより多くのエラーメッセージを知ることが出来ます.
  2. (roscd jsk_perception; export GIT_PAGER=cat; git branch -a; git diff; git rev-parse HEAD; cd -) として,プログラムのソースのバージョンがどいうものかわかります.
  3. 別のターミナルで rostopic echo /skeleton_with_depth/output/skeleton としておく.

@tkmtnt7000
Copy link
Member

tkmtnt7000 commented May 17, 2023

デバッグメモ:
手先のUSBハブに

  • 目玉デバイス
  • D435
  • Respeaker

を全て繋いでrealsenseのノードを立ち上げようとするとd435との接続が切れる
目玉デバイスもしくはRespeakerを外すと正常にrealsenseのノードをたちあげられる
3つ繋ぐと電源供給が足りなかった可能性が考えられる

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants