CaptainCook4D: A dataset for understanding errors in procedural activities

Rohith Peddi1, Shivvrat Arya1, Bharath Challa1, Likhitha Pallapothula1, Akshay Vyas1, Bhavya Gouripeddi1, Jikai Wang1, Qifan Zhang1, Vasundhara Komaragiri1, Nicholas Ruozzi1 Eric Ragan2 Yu Xiang1 Vibhav Giridhar Gogate1
1The University of Texas at Dallas 2The University of Florida
DMLR Workshop @ICML 2023 (Accepted),
Home
Overview Recipes 2D Data 3D Data

Recipe Information

S.No Recipe Name Heating Type Recipe Text
1 Microwave Egg Sandwich Microwave Coat a 6-oz. ramekin cup with cooking spray. Pour 1 egg into the ramekin cup. Microwave the ramekin cup uncovered on high for 30 seconds and stir. Continue to Microwave for 15-30 more seconds or until the egg is almost set. Top with 1 tablespoon of salsa and sprinkle with 1 tablespoon of cheese. Microwave just until cheese melts, about 10 seconds. Cut the English muffin into two pieces with a knife. Line the bottom of the English muffin with lettuce. Place the egg from the cup over the lettuce and replace the top of the English muffin.
2 Dressed Up Meatballs Microwave Cut 1/4 medium carrot into short, thin strips. Peel one medium onion and cut it into two pieces. Slice 1/8 medium onion. Peel 1 garlic clove. Cut and Mince 1/8 garlic clove. Place-Place 5 meatballs in a Microwave-safe plate. Top the plate with the carrots, onion, garlic and 1/4 tsp pepper powder. Mix 1/4 cup sweet-and-sour sauce and 1/2 teaspoon soy sauce in a small bowl. Pour the sauces over the meatballs. Microwave, covered, on high for 5 minutes Stir the contents in the microwave with a spoon Microwave, covered, on high for 5 minutes Stir the contents in the microwave with a spoon. Microwave for 1 more minute.
3 Microwave Mug Pizza Microwave Take a microwavable mug. Measure 4 tablespoons of flour and add it to the mug. Measure 1/8 teaspoon of baking powder and add it to the mug. Measure 1/16 teaspoon of baking soda and add it to the mug. Measure 1/8 teaspoon of salt and add it to the mug. Stir the contents in the mug well. Add in 3 tablespoons of milk and 1 tablespoon of olive oil to the mug. Mix the contents of the mug thoroughly(There might be some lumps, but that is ok). Take 1 tablespoon of marinara sauce and spread it around the surface of the batter. Sprinkle 1 generous tablespoon of mozzarella cheese on top of the sauce. Sprinkle dried Italian herbs inside the mug. Microwave for 1 minute 20 seconds, or until it rises and the toppings are bubbling.
4 Ramen Microwave Remove the noodles from the package(Break Noodles / Keep them as a block). Peel and Chop 1 garlic clove on a cutting board. Peel 1 medium onion and slice 1/4 medium onion into pieces. Put all the Vegetables in a microwave-safe bowl. Add the noodles to the bowl and cover them with water. Cover with a lid (or paper towel) to prevent splattering. Microwave the ramen for 4 minutes. Add basil and cilantro to the bowl. Let the noodles sit for about 1 minute after the microwave stops. Mix in the flavour packet. Stir noodles with a spoon or fork until the flavouring dissolves.
5 Coffee Nothing Measure 12 ounces of cold water and transfer it to a kettle. Boil the water. While the water is boiling, assemble the filter cone. Place the dripper on top of a coffee mug. Prepare the filter insert by folding the paper filter in half to create a semi-circle, and in half again to create a quarter-circle. Place the paper filter in the dripper and spread open to create a cone. Weigh the coffee beans (8 oz – 12 oz). Grind the coffee beans until the coffee grounds are the consistency of coarse sand, about 20 seconds. Transfer the grounds to the filter cone. Once the water has boiled, check the temperature of the water(The water should be between 195-205 degrees Fahrenheit or between 91-96 degrees Celsius). If the water is too hot, let it cool briefly. Pour a small amount of water into the filter to wet the grounds. Wait about 30 seconds for the coffee to bloom(You will see small bubbles or foam on the coffee grounds during this step). Slowly pour the rest of the water over the grounds in a circular motion. Do not overfill beyond the top of the paper filter. Let the coffee drain completely into the mug before removing the dripper. Discard the paper filter and coffee grounds.
6 Breakfast Burritos Microwave Extract the contents of an egg and add it to a microwave-safe bowl. Whisk the egg. Microwave for 3 minutes, stirring in between. Add 1 tbsp salsa to the bowl. Sprinkle oregano in the bowl. Add 1/2 tbsp sweet and sour sauce to the bowl. Mix the contents of the bowl well. Place 8 inch tortilla on a cutting board. Pour egg mixture on top of the tortilla. Sprinkle 1 tbsp shredded cheddar cheese on top of the egg. Roll the tortilla from one end to another into a log shape, about 1.5 inches thick. Roll it tight enough to prevent gaps but not so tight that the filling leaks.
7 Spiced Hot Chocolate Microwave Fill a microwave-safe mug with skimmed milk. Microwave the contents of the mug for 1 minute. Add 2 pieces of chocolate to the mug. Add a 1/5 teaspoon of cinnamon to the mug. Add 1 teaspoon of white sugar. Mix the contents of the mug. Heat the contents of the mug for 1 minute and serve.
8 Microwave French Toast Microwave In a large mug, melt 1 tablespoon of softened butter in the microwave for about 30 seconds. Roll the butter around in the cup to coat it. In the mug, whisk one egg with a fork until well blended. Sprinkle 1/4 teaspoon cinnamon over the egg. Add 1/4 teaspoon vanilla extract and stir again. Cut or tear 1 slices of bread into bite-size pieces and add to the egg mixture in the mug, pressing the bread down into the egg. Microwave on high for 90 seconds until the egg is cooked through. Put the cup's contents on a plate, cut it up, and serve.
9 Pinwheels Nothing Place 8-inch flour tortilla on cutting board. Use a butter knife to scoop nut butter from the jar. Spread nut butter onto the tortilla, leaving 1/2-inch uncovered at the edges. Clean the knife by wiping it with a paper towel. Use the knife to scoop jelly from the jar. Spread jelly over the nut butter. Clean the knife by wiping with a paper towel. Roll the tortilla from one end to the other into a log shape, about 1.5 inches thick. Roll it tight enough to prevent gaps, but not so tight that the filling leaks. Secure the rolled tortilla by inserting 5 toothpicks about 1 inch apart. Trim the ends of the tortilla roll with the butter knife, leaving ½ inch margin between the last toothpick and the end of the roll. Discard ends. Slide floss under the tortilla, perpendicular to the length of the roll. Place the floss halfway between two toothpicks. Cross the floss's two ends over the tortilla roll's top. Holding one end of the floss in each hand, pull the floss ends in opposite directions to slice. Continue slicing with floss to create 1 more pinwheel. Continue slicing with floss to create 1 more pinwheel. Continue slicing with floss to create 1 more pinwheel. Place the pinwheels on a plate.
10 Tomato Mozzarella Salad Nothing Rinse a tomato and gently dry it with a tea towel. Slice one tomato into about ½ inch thick slices. Place the thick slices of tomatoes on a platter, ensuring they only make a single layer. Season the tomato slices with salt. Sprinkle mozzarella cheese on top of the tomato throughout the platter. Garnish with italian seasoning. Season with ¼ teaspoon black pepper. Add a drizzle of extra-virgin olive oil, about 1 tablespoon, over the entire platter.
11 Butter Corn Cup Microwave Measure 2 cups of frozen corn. Thaw the frozen corn by putting it in a sieve and running it under cold water. Add the corn into a microwave-safe bowl. Microwave the corn for 2 minutes. Add 1 teaspoon of softened butter and 1 teaspoon of pepper powder to the bowl and then stir. Microwave the corn for 3 more minutes. Add 1 teaspoon salt to the bowl. Extract lime juice from 1/3 lime and add it to the bowl. Mix the contents of the bowl well.
12 Tomato Chutney Pan Peel 4 large garlic cloves. mince the garlic. Take 1 tomato. Chop them up roughly and puree them without any water in a blender/mixer. Heat 3 tbsp oil in a pan over medium heat. Add 1/4 tsp mustard and 1/2 tsp cumin seeds. When the seeds begin to sizzle, add minced garlic. Lower the heat. Saute the garlic for 2-3 minutes. Add 2 tbsp red chili powder and mix well. Add tomato puree and 1/2 tsp salt. Mix well. Allow the mixture to simmer over low heat for 5 minutes or until the mixture becomes thick. Take the pan off the heat. Transfer it to a serving bowl.
13 Scrambled Eggs Pan Chop 1/4 medium onion. Chop 1/4 tomato. Chop 1 green chili. Peel 2 garlic cloves. Mince peeled garlic cloves. Chop 1 tbsp cilantro. Heat 2 tbsp oil in a heavy-bottomed or nonstick pan on medium heat, and add chopped onions and 1/3 tsp salt. Sauté the onions on medium heat until they are soft and translucent. Add garlic, and chili to the pan. Cook for 1 minute, mixing everything. Add tomatoes and 1/8 tsp of turmeric to the pan. Cook covered for 1 minute or until the tomatoes soften. Whisk one egg with 1 tbsp milk and 1/3 tsp salt. Slowly pour the whisked eggs into the pan. Keep mixing with a spatula for 3 minutes or until the eggs are almost cooked. Garnish with 1 tbsp chopped cilantro and serve.
14 Cucumber Raita Nothing In a mixing bowl, whisk 1 cup of chilled curd until smooth. Use fresh homemade or packaged curd. Avoid using sour-tasting curd. Rinse, peel and chop or grate 1 medium-sized cucumber (You need 1 cup of chopped or grated cucumber). Add the chopped or grated cucumber to the whisked curd. Add 1 teaspoon of cumin powder, 1/2 teaspoon of chaat masala powder, 1/4 teaspoon of red chilli powder and 1/4 teaspoon salt. Lastly, add 1 tablespoon of chopped cilantro leaves Combine all the ingredients.
15 Zoodles Pan Spiralize 1 medium zucchini into thin noodles using a spiralizer. Heat a large pan on medium heat. Melt 1 tablespoons of softened butter. Peel 1 garlic cloves. Add 1 large minced garlic cloves to the pan. Cook garlic until fragrant (about 1 minutes). Be careful not to burn garlic. Add the zucchini noodles and 1/6 cup grated parmesan cheese and season with salt & pepper to taste. Cook for 1 minutes or until the zoodles are done. Remove from heat and serve immediately. Top with more parmesan if desired.
16 Sauted Mushrooms Pan Chop 1 shallot Rinse 3 mushrooms under cold water and pat them dry with a paper towel. Pull out the stems. Slice the mushrooms. Heat 1 tbsp olive oil in a large skillet over medium-high heat. Once the pan is hot, add the mushrooms and cook for 3 minutes, stirring often, until they start to soften and brown. Add chopped shallot and cook, often stirring, for 1 minute. Peel 2 garlic cloves and mince them. Add 2 cloves of minced garlic and 1/4 tbsp balsamic vinegar and cook, often stirring, for 1 minute. Season with salt and pepper to taste. Transfer the contents of the pan to a serving dish.
17 Blender Banana Pancakes Pan Add 1 banana, 1 egg, 1 heaped tbsp flour and 1/2 tsp baking powder to a blender and blitz for 20 seconds. Melt a small knob of butter in a non-stick frying pan over low-medium heat. Pour three little puddles straight from the blender into the frying pan. Cook for 1 min or until the tops start to bubble. Flip with a fork or a fish slice spatula and cook for 20-30 seconds more. Transfer to a plate. Chop 1 strawberry. Serve the pancakes with chopped strawberries and a splash of maple syrup if you like.
18 Herb Omelet with Fried Tomatoes Pan Take a tomato and cut it into two pieces. Heat 1 tbsp oil in a small non-stick frying pan, then cook the tomatoes cut-side down until they start to soften and colour. Chop 2 tbsp cilantro. Meanwhile, crack one egg in a bowl and add the chopped cilantro and 1/2 tsp ground black pepper. Beat the contents of the bowl. Scoop the tomatoes from the pan and put tomatoes on a serving plate. Pour the egg mixture into the pan and stir gently with a wooden spoon so the egg that sets on the base of the pan moves to enable the uncooked egg to flow into the space. Stop stirring when it’s nearly cooked to allow it to set into an omelette. Transfer it to the plate and serve with the tomatoes.
19 Broccoli Stir Fry Pan Peel 2 cloves of garlic and mince them. Combine 1/6 cup water, 1/8 cup soy sauce, 1 tablespoon honey, 2 cloves minced garlic, 1/2 tablespoon minced fresh ginger, 1/8 teaspoon black pepper, 1 teaspoon cornstarch in a small bowl. Whisk. Set aside. Take 5 in number broccoli florets. Take 2 cremini mushrooms, and slice them. Take 1 bell pepper and slice 1/3 of the bell pepper. Heat 2 tablespoons olive oil in a skillet over medium-high heat. Add the broccoli and sliced mushrooms and cook, stirring often, for 4 minutes. If the pan gets too hot on medium-high, turn the heat down to medium. Add the bell pepper and continue cooking, stirring often, for 2-3 minutes, until vegetables are crisp-tender. Whisk the sauce again to recombine the ingredients. Pour the sauce into the pan and cook, stirring, for 1 minute until the sauce thickens.
20 Pan Fried Tofu Pan Cut 1/4 block or 3 ounces of fresh tofu into large cubes and pat it dry with a towel. Add 1 tablespoon of olive oil to a non-stick pan and add the tofu cubes and a 1/4 tsp of salt. Turn on the heat to medium Cook 5 to 6 minutes until tofu cubes are lightly browned on the bottom. Briefly remove the pan from the heat to reduce spitting. Flip the tofu with tongs. Return the heat to medium and cook until browned. Briefly remove from the heat again and drizzle with the 1 tablespoons sesame oil and 1 tablespoon soy sauce (watch for spitting). Return to low heat and cook 2 minutes, then flip and cook 2 minutes until the colour is darkened. Transfer to a serving dish
21 Mug Cake Microwave Place the paper cupcake liner inside the mug. Set aside. Measure and add 2 tbsp flour, 1.5 tbsp sugar, 1/4 tsp baking powder, and a pinch of salt to the mixing bowl. Whisk to combine. Measure and add 2 tsp vegetable oil, 2 tbsp water, and 1/4 tsp vanilla extract to the bowl. Whisk batter until no lumps remain. Pour batter into prepared mug. Microwave the mug and batter on high power for 60 seconds. Check if the cake is done by inserting and toothpick into the center of the cake and then removing it. If wet batter clings to the toothpick, microwave for an additional 5 seconds. If the toothpick comes out clean, continue. Invert the mug to release the cake onto a plate. Allow to cool until it is no longer hot to the touch, then carefully remove the paper liner. While the cake is cooling, prepare to pipe the frosting. Scoop 4 spoonfuls of chocolate frosting into a zip-top bag and seal, removing as much air as possible. Use scissors to cut one corner from the bag to create a small opening ¼-inch in diameter. Squeeze the frosting through the opening to apply small dollops of frosting to the plate in a circle around the base of the cake.
22 Cheese Pimiento Microwave Chop 1/4 red bell pepper into tiny bits. Place the chopped pepper in a microwave-safe bowl. Add 1 tablespoons of water to the bowl. Microwave-Microwave the bowl, covered, for 2 minutes. Add 1/3 cup cheddar cheese to a microwave-safe cup. Melt the cheese by microwaving it for 30 sec. (Check after 30 seconds and microwave for 10 seconds more if needed). Mix the cheese and red bell pepper in the bowl and add 1/2 tbsp softened butter. Add 1/4 teaspoon salt. Add 1/4 teaspoon pepper. Mix-Mix all the ingredients of the bowl well.
23 Spicy Tuna Avocado Wraps Nothing Take 1 ripe avocado and cut it into thin slices. Chop 1 scallion. Open a can of tuna and drain excess water. Add 1 can drained tuna, 1/4 cup mayonnaise, 1 tsp Sriracha sauce, and chopped scallion in a bowl. Mix the contents of the bowl. Season with 1/4 tsp salt and 1/4 tsp pepper. Lay out 2 large lettuce leaves, place avocado slices on each leaf, and top with the tuna mixture. Roll up the lettuce wraps and secure wrap with a toothpick.
24 Caprese Bruschetta Pan Slice 2 1/2 inch thick rounds from a baguette (slice slanted) Brush 2 slices of baguette with olive oil on both sides. Toast both sides of the slices on the pan for 2 to 3 minutes until lightly charred and crispy. Cut 1/4 cup of cherry tomatoes into halves. In a bowl, add the cut cherry tomatoes, 1/8 cup shredded mozzarella, 1/16 cup basil, 1/4 tsp salt, and 1/4 tsp pepper. Combine the contents of the bowl. Spoon the mixture onto the bread.

Task graphs for all selected recipes

Loading...