Computer and Robot Vision

Practical Assignment

Background and arrangements

The students need to develop a robotic system to solve a simple problem (e.g. solve a simple puzzle, stack some blocks…). The assignment will be done in groups of around 6 people. The groups will be chosen by the lecturer of the course. The groups are shown on the Noppa page of the course. If you are not a member in any group, but want to complete the assignment, please contact the lecturer (make sure you collect SOME points in the 2nd and 3rd exercise to show that you are participating in the course - inactive students will be left out from the groups).

Required steps

  1. Each group will prepare an initial plan for the assignment. The plan will outline the setting (for example, which puzzle), the used hardware, and propose an approach.
    • The plan should include the list of the names of the students who have contributed to the plan.
    • Deadline for the plans is the exercise session on Oct 24th 2012.
  2. Each plan is commented on by another student group and the lecturer. These comments are meant to raise possible questions already before any implementation work is begun.
  3. Each group has the opportunity to visit the lab with the robot to check that their plan is physically viable.
    • Robot motion area as well as camera placement can be checked.
    • Test images can also be taken at this point.
    • The visits will take place in October
    • Each group needs to agree on their time slot (1 hour) with Dr Jarmo Ilonen. Please contact Jarmo by email.
  4. The student groups will perform the development and implementation work.
    • Access to the lab can be got by agreeing with lab personnel (Dr Jarmo Ilonen).
  5. Each group needs to submit a progress report by December 20th 2012 (by e-mail).
    • If the group's project work is ready, the students can deliver the final report and demonstrate their system instead.
    • A progress report should briefly describe the work done so far and outline a schedule for the remaining work.
  6. Final deadline for the project (report and demonstration) is December 14th 2012 moved to January due to delays in getting to use the robot


To pass the practical assignment, each group has to:

  • deliver an initial plan by deadline given above
  • deliver a progress report by deadline given above
  • demonstrate that their system can solve the puzzle as described by the initial plan
  • deliver a final report describing the problem and its solution.
    • All methods used in the solution need to be documented. References to literature can be used when applicable (a citation is sufficient for methods that are taken from the literature).

Each group member must contribute to the effort. When delivering the final report, the contributing group members are asked to sign the final report. By signing the report, group members certify that each member named on the report has a sufficient personal contribution.

MELFA RV3-SB, a 6 DOF robot arm and the attached Robotiq gripper will be used. Low-level interfaces to the robot arm and gripper exist (see below for Available interfaces), but also a binary for controlling the robot will be provided. The inter-process communication with the binary is done by reading and writing normal files, so using it from for example Matlab is easy.

Vision needs to be used in order to locate the puzzle pieces. Kinect RGB-D camera will be used. It provides a normal RGB image as well as depth image. A binary for capturing images will be provided. The binary also generates a point cloud and the students can choose which information the project will use.

The software needs to operate on the robot control computer running Ubuntu 10.04. Additional software can be installed on the computer. Any programming language or a combination thereof available at the computer can be used, but using Matlab is encouraged.

Matlab toolboxes available at the university as well as any freely available software libraries can be used. All methods used (also those from the libraries) need to be documented, that is, the methods used by the external libraries need to be documented.

Available interfaces

Interfaces to Kinect and robot/gripper will be provided as binaries. The documentation, some example Matlab scripts and example images are provided here.

Hardware use

Each student group gets a user account on the robot controller computer The computer is mainly used as research equipment and thus students need to be extra careful in the use of the equipment.

Rules and information for using

  • Students are required to exercise care when using the computer.
  • The computer can only be used for developing the practical assignment. No other use is allowed.
  • The computer can be used locally or remotely (via ssh).
    • Because of the equipment's research use, students can be asked to log off from the computer for a period, if the use would hinder research use.
  • Software using any local hardware (camera, robot, etc.) must only be run locally on the console. Don't try to use hardware using a remote connection.
  • Each student group is responsible for the user account they have received.
    • The group is responsible for keeping the password safe.
    • Software can be installed under group's own account.
  • Robot hardware can only be tested when a member of staff is present.
  • In any problem situation, a staff member needs to be contacted.
    • Students are not allowed to attempt to solve occurring problems themselves.
  • No changes to the computer environment are allowed without permission from the lecturer.
  • Necessary software can be installed by staff if requested by students.

Documentation and Software

If you require some more documentation, please contact the lecturer by email.