KITchen benchmark v1.0

Benchmark

Frozen splits, public ground truth and one command that checks, scores and writes a citable result card for 2D detection, instance segmentation and 6D pose.

Take part See the leaderboard

terminal

$ pip install git+https://github.com/KITchen-Dataset/KITchenBenchmark@v1.0
$ kitchenbench verify kitchen_bench_v1.0
KITchen benchmark 1.0 OK (539 files, ground truth 665635036314)
$ kitchenbench eval kitchen_bench_v1.0 pose my_poses.csv --method method.json
AP_6D  AP_MSSD  AP_MSPD  AR_6D  fps  rules met
wrote results/my-method_pose_test.json  .md  .tex

Scored test poses: 377,846
Scored test boxes and masks: 443,898
Object models scored: 107
Object classes: 108
Tasks: 3
Pose speed rule: ≥ 5 fps

Three tasks

What you can be scored on

Each task uses the community's standard protocol, so existing methods and evaluation code plug in unchanged.

2D object detection

COCO

You predict: A box, class and score for each object, as a COCO results file.
Primary score: AP averaged over IoU 0.50 to 0.95
Also reported: AP50, AP75, AP for small, medium and large objects, AR at 300 detections

Instance segmentation

COCO

You predict: An RLE mask, class and score for each object, as a COCO results file.
Primary score: Mask AP averaged over IoU 0.50 to 0.95
Also reported: AP50, AP75, AP by object size, AR at 300 detections

6D object pose

BOP

You predict: Rotation, translation in mm, object id, score and run time per image, as a BOP CSV.
Primary score: AP_6D, the mean of AP_MSSD and AP_MSPD (BOP 2024)
Also reported: AR_6D (BOP 2019). Symmetric objects are scored with their symmetries.

Results on the test split

Leaderboard

Ranked by the primary score. Every row links to its result card, which stores the checksums needed to reproduce the number.

#	Method	Input	Training data	AP	AP50	AP75	Card

No detection or segmentation results yet

Score your method with one command and open an issue with its result card to appear here first.

How to submit

#	Method	Training data	AP_6D	AP_MSSD	AP_MSPD	AR_6D	fps	Rules	Card

No RGB pose results yet

Score your method with one command and open an issue with its result card to appear here first.

How to submit

Detection and segmentation methods are ranked together whatever their input; the card states it.

Take part

From predictions to the leaderboard

The same command checks your file, scores it and writes the card you cite, so a reported number can always be traced back to the exact ground truth and predictions.

Get the benchmark
Install the evaluator and download the v1.0 files. verify confirms your copy of the ground truth is the frozen one.
Train and tune
Train on train and tune on val. No test frame or test annotation may be used.
Predict on the test frames
Run on every test frame (splits/test_eval.txt): a COCO results file for detection and segmentation, a BOP CSV with the run time of each image for pose.
Score and submit
validate catches format mistakes, eval writes the result card with a ready-made table row for your paper. Open an issue on the benchmark repository with the card attached.

take_part.sh

# 1. get the benchmark
pip install git+https://github.com/KITchen-Dataset/KITchenBenchmark@v1.0
unzip kitchen_bench_v1.0.zip
kitchenbench verify kitchen_bench_v1.0

# 3. describe your method once
kitchenbench method-template > method.json

# 4. check and score (det, segm or pose)
kitchenbench validate kitchen_bench_v1.0 pose my_poses.csv
kitchenbench eval     kitchen_bench_v1.0 pose my_poses.csv --method method.json
# results/my-method_pose_test.json  .md  .tex

Fair comparison

Rules and splits

Rules

Report the version with every number, for example "KITchen v1.0 AP_6D".
State the input and all training data, pretraining included. Pose methods using RGB and RGB-D are ranked separately.
One model for all objects for pose, as in the KITchen paper.
At least 5 fps for pose, measured as wall time per image on the hardware you state.
Pose methods that meet both paper rules rank first. The others are listed below them.

How the splits are made

Train is one room, val and test another. KITchen was recorded in an open lab room and in a built-in kitchen. Methods train in the lab room and are scored in the kitchen, a room they have never seen.
Whole setups go to val or test. A setup is one arrangement of objects, and it recurs across many short clips. Kitchen clips are grouped by arrangement (distances between objects seen together match to within a few centimetres), and each setup goes whole to val or test, so no arrangement is in both.
Every test frame is scored, so a result covers every pose and box in the test split.
The test ground truth is public and the evaluator is frozen and versioned. New annotations always mean a new version.

See the splits on the Explore page

Get the benchmark

Download & cite

Benchmark files v1.0

One archive with everything the evaluator needs. The images come from the dataset download.

Splits train, validation and test frame lists, and the setup of every frame
2D ground truth COCO files with boxes, masks and ignore regions
6D ground truth BOP files with poses and the pairs left out of scoring
Evaluation models with their symmetries
Manifest benchmark.json with the checksum of every file

Ground truth checksum 665635036314 · archive 327.2 MB, sha256 8e5373bc185352a91ddcf72470a2bb8da4efd66cba2ffa7f1eb44af188395c7d

Download v1.0 Code on GitHub Design and rules

Cite the KITchen paper and state the benchmark version

@inproceedings{younes2024kitchen,
  title     = {{KITchen}: A Real-World Benchmark and Dataset
               for {6D} Object Pose Estimation in Kitchen Environments},
  author    = {Younes, Abdelrahman and Asfour, Tamim},
  booktitle = {2024 IEEE-RAS 23rd International Conference
               on Humanoid Robots (Humanoids)},
  pages     = {803--810},
  year      = {2024},
  doi       = {10.1109/Humanoids58906.2024.10769947},
}