CaptainCook4D: A dataset for understanding errors in procedural activities

Rohith Peddi1, Shivvrat Arya1, Bharath Challa1, Likhitha Pallapothula1, Akshay Vyas1, Jikai Wang1, Qifan Zhang1, Vasundhara Komaragiri1, Nicholas Ruozzi1 Eric Ragan2 Yu Xiang1 Vibhav Giridhar Gogate1
1The University of Texas at Dallas 2The University of Florida
DMLR Workshop @ICML 2023 (Accepted),
Home
Overview Recipes 2D Data 3D Data

Overview

S.No Recipe Name Recording Id Person Id Environment Id Video GoPro Video Hololens Depth Spatial Audio
1 Pinwheels 10_16 5 2
2 Pinwheels 10_18 2 1
3 Pinwheels 10_24 5 11
4 Pinwheels 10_26 2 6
5 Pinwheels 10_31 8 2
6 Pinwheels 10_42 8 5
7 Pinwheels 10_46 4 5
8 Pinwheels 10_47 4 7
9 Pinwheels 10_48 5 1
10 Pinwheels 10_50 2 5
11 Pinwheels 10_6 8 7
12 Pinwheels 10_7 8 6
13 Tomato Mozzarella Salad 12_10 3 2
14 Tomato Mozzarella Salad 12_119 5 11
15 Tomato Mozzarella Salad 12_12 5 1
16 Tomato Mozzarella Salad 12_13 4 5
17 Tomato Mozzarella Salad 12_15 2 2
18 Tomato Mozzarella Salad 12_16 3 6
19 Tomato Mozzarella Salad 12_17 2 6
20 Tomato Mozzarella Salad 12_19 3 6
21 Tomato Mozzarella Salad 12_2 4 1
22 Tomato Mozzarella Salad 12_26 2 1
23 Tomato Mozzarella Salad 12_38 5 5
24 Tomato Mozzarella Salad 12_41 5 7
25 Tomato Mozzarella Salad 12_43 8 5
26 Tomato Mozzarella Salad 12_48 8 2
27 Tomato Mozzarella Salad 12_5 2 5
28 Tomato Mozzarella Salad 12_51 8 7
29 Tomato Mozzarella Salad 12_6 3 2
30 Tomato Mozzarella Salad 12_9 3 5
31 Butter Corn Cup 13_12 2 2
32 Butter Corn Cup 13_14 8 2
33 Butter Corn Cup 13_18 5 1
34 Butter Corn Cup 13_20 3 1
35 Butter Corn Cup 13_24 2 6
36 Butter Corn Cup 13_31 8 1
37 Butter Corn Cup 13_32 8 6
38 Butter Corn Cup 13_36 5 11
39 Butter Corn Cup 13_38 8 5
40 Butter Corn Cup 13_41 4 7
41 Butter Corn Cup 13_44 4 6
42 Butter Corn Cup 13_45 2 1
43 Butter Corn Cup 13_5 5 2
44 Butter Corn Cup 13_9 2 5
45 Tomato Chutney 15_17 4 1
46 Tomato Chutney 15_18 4 7
47 Tomato Chutney 15_19 3 1
48 Tomato Chutney 15_2 8 6
49 Tomato Chutney 15_28 5 2
50 Tomato Chutney 15_29 2 2
51 Tomato Chutney 15_30 4 6
52 Tomato Chutney 15_33 8 1
53 Tomato Chutney 15_37 4 2
54 Tomato Chutney 15_39 5 11
55 Tomato Chutney 15_4 8 2
56 Tomato Chutney 15_41 4 5
57 Tomato Chutney 15_46 1 7
58 Tomato Chutney 15_5 5 5
59 Tomato Chutney 15_8 3 5
60 Scrambled Eggs 16_1 7 5
61 Scrambled Eggs 16_10 6 10
62 Scrambled Eggs 16_17 4 8
63 Scrambled Eggs 16_18 6 2
64 Scrambled Eggs 16_2 6 5
65 Scrambled Eggs 16_20 1 8
66 Scrambled Eggs 16_23 7 2
67 Scrambled Eggs 16_26 7 6
68 Scrambled Eggs 16_27 7 9
69 Scrambled Eggs 16_28 7 1
70 Scrambled Eggs 16_3 3 8
71 Scrambled Eggs 16_35 1 1
72 Scrambled Eggs 16_39 6 7
73 Scrambled Eggs 16_40 3 1
74 Scrambled Eggs 16_42 1 2
75 Scrambled Eggs 16_44 6 1
76 Cucumber Raita 17_10 4 6
77 Cucumber Raita 17_11 3 2
78 Cucumber Raita 17_14 8 7
79 Cucumber Raita 17_15 2 6
80 Cucumber Raita 17_16 5 11
81 Cucumber Raita 17_19 2 2
82 Cucumber Raita 17_20 5 5
83 Cucumber Raita 17_21 2 5
84 Cucumber Raita 17_23 8 6
85 Cucumber Raita 17_28 3 8
86 Cucumber Raita 17_29 5 7
87 Cucumber Raita 17_3 2 1
88 Cucumber Raita 17_36 4 8
89 Cucumber Raita 17_37 5 1
90 Cucumber Raita 17_40 4 7
91 Cucumber Raita 17_43 1 3
92 Cucumber Raita 17_45 4 2
93 Cucumber Raita 17_49 4 1
94 Cucumber Raita 17_5 8 5
95 Cucumber Raita 17_8 5 2
96 Zoodles 18_101 8 6
97 Zoodles 18_11 4 2
98 Zoodles 18_12 3 3
99 Zoodles 18_19 4 2
100 Zoodles 18_2 8 2
101 Zoodles 18_24 3 2
102 Zoodles 18_27 1 5
103 Zoodles 18_28 1 2
104 Zoodles 18_3 1 1
105 Zoodles 18_31 4 5
106 Zoodles 18_33 4 6
107 Zoodles 18_41 3 6
108 Zoodles 18_45 5 11
109 Zoodles 18_49 5 2
110 Zoodles 18_8 1 3
111 Microwave Egg Sandwich 1_10 6 1
112 Microwave Egg Sandwich 1_136 3 1
113 Microwave Egg Sandwich 1_14 1 5
114 Microwave Egg Sandwich 1_143 7 1
115 Microwave Egg Sandwich 1_19 1 7
116 Microwave Egg Sandwich 1_20 1 2
117 Microwave Egg Sandwich 1_25 6 7
118 Microwave Egg Sandwich 1_28 1 1
119 Microwave Egg Sandwich 1_30 6 5
120 Microwave Egg Sandwich 1_32 7 9
121 Microwave Egg Sandwich 1_33 1 6
122 Microwave Egg Sandwich 1_34 6 10
123 Microwave Egg Sandwich 1_36 3 1
124 Microwave Egg Sandwich 1_37 6 2
125 Microwave Egg Sandwich 1_42 7 5
126 Microwave Egg Sandwich 1_43 7 1
127 Microwave Egg Sandwich 1_49 7 6
128 Microwave Egg Sandwich 1_7 3 8
129 Sauted Mushrooms 20_14 5 5
130 Sauted Mushrooms 20_16 4 2
131 Sauted Mushrooms 20_17 5 7
132 Sauted Mushrooms 20_19 8 2
133 Sauted Mushrooms 20_22 2 1
134 Sauted Mushrooms 20_25 8 7
135 Sauted Mushrooms 20_26 4 1
136 Sauted Mushrooms 20_29 4 5
137 Sauted Mushrooms 20_32 8 5
138 Sauted Mushrooms 20_39 3 6
139 Sauted Mushrooms 20_44 2 6
140 Sauted Mushrooms 20_47 3 2
141 Sauted Mushrooms 20_48 2 2
142 Sauted Mushrooms 20_9 3 5
143 Blender Banana Pancakes 21_103 1 1
144 Blender Banana Pancakes 21_11 1 5
145 Blender Banana Pancakes 21_14 7 2
146 Blender Banana Pancakes 21_15 7 5
147 Blender Banana Pancakes 21_17 6 10
148 Blender Banana Pancakes 21_19 1 7
149 Blender Banana Pancakes 21_24 7 9
150 Blender Banana Pancakes 21_25 7 1
151 Blender Banana Pancakes 21_28 3 6
152 Blender Banana Pancakes 21_29 7 6
153 Blender Banana Pancakes 21_3 1 1
154 Blender Banana Pancakes 21_32 3 6
155 Blender Banana Pancakes 21_37 6 5
156 Blender Banana Pancakes 21_43 1 6
157 Blender Banana Pancakes 21_44 3 5
158 Blender Banana Pancakes 21_46 6 2
159 Blender Banana Pancakes 21_47 1 2
160 Blender Banana Pancakes 21_50 6 1
161 Blender Banana Pancakes 21_8 3 1
162 Herb Omelet with Fried Tomatoes 22_10 1 1
163 Herb Omelet with Fried Tomatoes 22_13 6 5
164 Herb Omelet with Fried Tomatoes 22_137 3 6
165 Herb Omelet with Fried Tomatoes 22_2 3 2
166 Herb Omelet with Fried Tomatoes 22_21 6 2
167 Herb Omelet with Fried Tomatoes 22_24 1 5
168 Herb Omelet with Fried Tomatoes 22_26 6 1
169 Herb Omelet with Fried Tomatoes 22_30 6 10
170 Herb Omelet with Fried Tomatoes 22_31 3 6
171 Herb Omelet with Fried Tomatoes 22_32 1 6
172 Herb Omelet with Fried Tomatoes 22_37 3 6
173 Herb Omelet with Fried Tomatoes 22_38 7 1
174 Herb Omelet with Fried Tomatoes 22_4 1 7
175 Herb Omelet with Fried Tomatoes 22_40 7 5
176 Herb Omelet with Fried Tomatoes 22_41 6 7
177 Herb Omelet with Fried Tomatoes 22_6 7 6
178 Herb Omelet with Fried Tomatoes 22_8 7 9
179 Broccoli Stir Fry 23_1 4 5
180 Broccoli Stir Fry 23_12 1 7
181 Broccoli Stir Fry 23_16 6 7
182 Broccoli Stir Fry 23_17 2 5
183 Broccoli Stir Fry 23_18 1 6
184 Broccoli Stir Fry 23_19 8 5
185 Broccoli Stir Fry 23_23 8 1
186 Broccoli Stir Fry 23_24 3 2
187 Broccoli Stir Fry 23_26 8 6
188 Broccoli Stir Fry 23_32 1 2
189 Broccoli Stir Fry 23_38 1 3
190 Broccoli Stir Fry 23_39 3 1
191 Broccoli Stir Fry 23_41 4 8
192 Broccoli Stir Fry 23_5 4 7
193 Broccoli Stir Fry 23_6 5 7
194 Broccoli Stir Fry 23_9 5 11
195 Pan Fried Tofu 25_1 4 6
196 Pan Fried Tofu 25_109 5 7
197 Pan Fried Tofu 25_11 4 2
198 Pan Fried Tofu 25_13 1 1
199 Pan Fried Tofu 25_2 1 2
200 Pan Fried Tofu 25_22 4 5
201 Pan Fried Tofu 25_3 8 1
202 Pan Fried Tofu 25_4 8 6
203 Pan Fried Tofu 25_40 2 2
204 Pan Fried Tofu 25_41 1 7
205 Pan Fried Tofu 25_42 1 6
206 Pan Fried Tofu 25_48 2 5
207 Pan Fried Tofu 25_5 4 7
208 Pan Fried Tofu 25_6 1 5
209 Pan Fried Tofu 25_9 2 6
210 Mug Cake 26_136 2 6
211 Mug Cake 26_17 5 5
212 Mug Cake 26_18 4 7
213 Mug Cake 26_19 5 7
214 Mug Cake 26_20 4 8
215 Mug Cake 26_22 3 3
216 Mug Cake 26_24 1 6
217 Mug Cake 26_29 2 5
218 Mug Cake 26_30 5 2
219 Mug Cake 26_34 4 6
220 Mug Cake 26_36 6 7
221 Mug Cake 26_39 8 7
222 Mug Cake 26_4 4 7
223 Mug Cake 26_42 8 6
224 Mug Cake 26_46 1 7
225 Mug Cake 26_6 3 8
226 Mug Cake 26_7 1 3
227 Cheese Pimiento 27_13 5 7
228 Cheese Pimiento 27_15 2 2
229 Cheese Pimiento 27_17 4 1
230 Cheese Pimiento 27_18 3 6
231 Cheese Pimiento 27_26 8 6
232 Cheese Pimiento 27_29 5 1
233 Cheese Pimiento 27_3 8 5
234 Cheese Pimiento 27_31 4 7
235 Cheese Pimiento 27_34 2 1
236 Cheese Pimiento 27_37 2 5
237 Cheese Pimiento 27_38 4 6
238 Cheese Pimiento 27_4 2 6
239 Cheese Pimiento 27_45 5 2
240 Cheese Pimiento 27_49 8 7
241 Cheese Pimiento 27_9 4 2
242 Spicy Tuna Avocado Wraps 28_10 6 1
243 Spicy Tuna Avocado Wraps 28_14 3 5
244 Spicy Tuna Avocado Wraps 28_16 3 3
245 Spicy Tuna Avocado Wraps 28_2 6 10
246 Spicy Tuna Avocado Wraps 28_21 7 9
247 Spicy Tuna Avocado Wraps 28_24 7 6
248 Spicy Tuna Avocado Wraps 28_25 1 2
249 Spicy Tuna Avocado Wraps 28_26 1 5
250 Spicy Tuna Avocado Wraps 28_28 7 1
251 Spicy Tuna Avocado Wraps 28_29 7 5
252 Spicy Tuna Avocado Wraps 28_3 6 7
253 Spicy Tuna Avocado Wraps 28_38 3 6
254 Spicy Tuna Avocado Wraps 28_42 1 1
255 Spicy Tuna Avocado Wraps 28_44 7 2
256 Spicy Tuna Avocado Wraps 28_45 6 2
257 Spicy Tuna Avocado Wraps 28_49 3 2
258 Spicy Tuna Avocado Wraps 28_50 3 1
259 Spicy Tuna Avocado Wraps 28_7 6 5
260 Caprese Bruschetta 29_129 1 7
261 Caprese Bruschetta 29_15 3 3
262 Caprese Bruschetta 29_17 4 8
263 Caprese Bruschetta 29_18 2 1
264 Caprese Bruschetta 29_19 3 8
265 Caprese Bruschetta 29_22 1 3
266 Caprese Bruschetta 29_28 2 6
267 Caprese Bruschetta 29_29 8 7
268 Caprese Bruschetta 29_32 5 5
269 Caprese Bruschetta 29_34 4 6
270 Caprese Bruschetta 29_35 5 7
271 Caprese Bruschetta 29_37 6 7
272 Caprese Bruschetta 29_45 5 1
273 Caprese Bruschetta 29_48 8 2
274 Caprese Bruschetta 29_49 4 2
275 Caprese Bruschetta 29_5 8 5
276 Caprese Bruschetta 29_6 5 2
277 Caprese Bruschetta 29_7 4 5
278 Dressed Up Meatballs 2_13 6 1
279 Dressed Up Meatballs 2_17 3 3
280 Dressed Up Meatballs 2_19 7 2
281 Dressed Up Meatballs 2_22 6 5
282 Dressed Up Meatballs 2_26 3 1
283 Dressed Up Meatballs 2_28 3 5
284 Dressed Up Meatballs 2_3 7 1
285 Dressed Up Meatballs 2_30 6 10
286 Dressed Up Meatballs 2_38 3 2
287 Dressed Up Meatballs 2_4 7 5
288 Dressed Up Meatballs 2_41 1 2
289 Dressed Up Meatballs 2_42 6 2
290 Dressed Up Meatballs 2_46 6 7
291 Dressed Up Meatballs 2_47 7 9
292 Dressed Up Meatballs 2_5 7 6
293 Dressed Up Meatballs 2_8 1 6
294 Microwave Mug Pizza 3_11 8 1
295 Microwave Mug Pizza 3_12 8 7
296 Microwave Mug Pizza 3_13 8 6
297 Microwave Mug Pizza 3_14 2 2
298 Microwave Mug Pizza 3_18 4 6
299 Microwave Mug Pizza 3_2 2 5
300 Microwave Mug Pizza 3_22 4 1
301 Microwave Mug Pizza 3_34 4 7
302 Microwave Mug Pizza 3_36 4 5
303 Microwave Mug Pizza 3_46 4 2
304 Microwave Mug Pizza 3_49 5 2
305 Microwave Mug Pizza 3_5 8 2
306 Microwave Mug Pizza 3_50 2 6
307 Ramen 4_122 3 6
308 Ramen 4_17 7 5
309 Ramen 4_2 6 2
310 Ramen 4_20 6 10
311 Ramen 4_22 3 1
312 Ramen 4_24 1 2
313 Ramen 4_3 7 6
314 Ramen 4_30 1 1
315 Ramen 4_32 1 6
316 Ramen 4_35 6 1
317 Ramen 4_36 3 2
318 Ramen 4_40 7 2
319 Ramen 4_43 1 7
320 Ramen 4_44 6 5
321 Ramen 4_5 7 9
322 Ramen 4_7 6 2
323 Ramen 4_9 7 1
324 Coffee 5_11 8 7
325 Coffee 5_15 3 6
326 Coffee 5_18 5 11
327 Coffee 5_19 4 7
328 Coffee 5_2 6 10
329 Coffee 5_22 6 7
330 Coffee 5_24 8 5
331 Coffee 5_27 5 5
332 Coffee 5_28 7 9
333 Coffee 5_3 7 5
334 Coffee 5_35 6 5
335 Coffee 5_37 7 1
336 Coffee 5_4 4 6
337 Coffee 5_42 3 5
338 Coffee 5_44 5 7
339 Breakfast Burritos 7_1 7 2
340 Breakfast Burritos 7_135 1 1
341 Breakfast Burritos 7_18 7 6
342 Breakfast Burritos 7_19 1 6
343 Breakfast Burritos 7_2 4 8
344 Breakfast Burritos 7_24 3 5
345 Breakfast Burritos 7_26 7 9
346 Breakfast Burritos 7_3 1 8
347 Breakfast Burritos 7_30 6 10
348 Breakfast Burritos 7_32 7 5
349 Breakfast Burritos 7_35 1 1
350 Breakfast Burritos 7_38 6 1
351 Breakfast Burritos 7_44 6 5
352 Breakfast Burritos 7_48 6 2
353 Breakfast Burritos 7_5 7 1
354 Breakfast Burritos 7_50 3 2
355 Spiced Hot Chocolate 8_11 3 6
356 Spiced Hot Chocolate 8_15 4 2
357 Spiced Hot Chocolate 8_16 4 6
358 Spiced Hot Chocolate 8_19 4 5
359 Spiced Hot Chocolate 8_20 5 2
360 Spiced Hot Chocolate 8_25 3 3
361 Spiced Hot Chocolate 8_26 8 5
362 Spiced Hot Chocolate 8_3 4 2
363 Spiced Hot Chocolate 8_30 5 5
364 Spiced Hot Chocolate 8_31 5 7
365 Spiced Hot Chocolate 8_33 5 11
366 Spiced Hot Chocolate 8_35 3 5
367 Spiced Hot Chocolate 8_40 2 5
368 Spiced Hot Chocolate 8_44 4 1
369 Spiced Hot Chocolate 8_45 8 2
370 Spiced Hot Chocolate 8_50 8 7
371 Microwave French Toast 9_108 7 2
372 Microwave French Toast 9_12 6 2
373 Microwave French Toast 9_13 6 1
374 Microwave French Toast 9_15 1 6
375 Microwave French Toast 9_19 1 2
376 Microwave French Toast 9_2 6 5
377 Microwave French Toast 9_22 1 7
378 Microwave French Toast 9_24 6 7
379 Microwave French Toast 9_25 3 1
380 Microwave French Toast 9_36 7 5
381 Microwave French Toast 9_4 1 5
382 Microwave French Toast 9_45 6 10
383 Microwave French Toast 9_47 7 6
384 Microwave French Toast 9_8 7 9