EgoEdit Logo

EgoEdit

Dataset, Real-Time Streaming Model, and Benchmark for Egocentric Video Editing

Runjia Li13, Moayed Haji Ali12, Ashkan Mirzaei1, Chaoyang Wang1, Arpit Sahni1, Ivan Skorokhodov1, Aliaksandr Siarohin1, Tomas Jakab3, Junlin Han3, Sergey Tulyakov1, Philip Torr3, Willi Menapace1,
1Snap Research 2Rice University 3University of Oxford
Paper Icon Paper Benchmark Icon EgoEditData (Under Final Review) Code Icon Code

We propose a framework for real-time egocentric video editing. Our system is composed of: EgoEditData, a manually curated dataset of 100k video editing pairs focusing on the egocentric case and featuring object substitution and removal under challenging hand occlusions, interactions, and large egomotion; EgoEdit the first real-time autoregressive model for egocentric video editing running in real time on a single H100 with 855ms first-frame latency and enabling live augmented reality (AR) interactions; EgoEditBench, a comprehensive benchmark for evaluation of egocentric video editing systems.

In-the-wild Egocentric Editing
(Generated by EgoEdit-RT )



















In-the-wild General Editing
(Generated by EgoEdit-RT)










Streaming Longer Videos
(Generated by EgoEdit-RT)

Baseline Comparison
(on samples from EgoEditBench)

Original Video
EgoEdit result 1
LucyEdit result 2
✏️ Edit Instruction
Instruction: Morph the white shaker bottle with blue cap into an ornate silver goblet — a medium-sized chalice with a rounded bowl and decorative filigreed stem, its polished metal rim and banding catching the light; the bowl glows an intense electric blue from within while the outer rim and upper surface are rimed in white frost and feathery ice crystals, with a thin, swirling freezing mist clinging to and spilling over the edges.
Original Video
EgoEdit result 1
LucyEdit result 2
✏️ Edit Instruction
Instruction: Replace the kitchen with a small home office desk with a laptop and a desk lamp.
Original Video
EgoEdit result 1
LucyEdit result 2
✏️ Edit Instruction
Instruction: Turn the depth map into a video with the following description: the egocentric subject cooks at a black stove in a kitchen with dark counters and gray walls, daylight from a right window. They hold a blue-capped shaker and a wooden spoon, sprinkling seasoning into red tomato sauce in a frying pan. A covered pot simmers and a cutting board with a chef's knife, chopped green peppers, pasta and a plastic bag sit nearby. The chest-mounted camera tilts as hands move.
Original Video
EgoEdit result 1
LucyEdit result 2
✏️ Edit Instruction
Instruction: Turn the video into a depth map
Original Video
EgoEdit result 1
LucyEdit result 2
✏️ Edit Instruction
Instruction: Apply ukiyo-e woodblock print art style to the video
Original Video
EgoEdit result 1
LucyEdit result 2
✏️ Edit Instruction
Instruction: Transform the bright green fabric with leaf print into a scarf in pale beige/cream—narrow and lightweight, draping into soft, flowing folds; the fabric should look smooth and silky with a subtle satin-like sheen and slight translucency, showing delicate creases and soft shadowing along the gathered waves.
Original Video
EgoEdit result 1
LucyEdit result 2
✏️ Edit Instruction
Instruction: Change the lounge to a quiet library with tall bookshelves and a warm reading lamp.
Original Video
EgoEdit result 1
LucyEdit result 2
✏️ Edit Instruction
Instruction: Replace the paperback held open in both hands with a magazine with a white, glossy cover featuring a dark rectangular photo panel near the top and a small printed logo/text centered near the bottom; make it spiral-bound along the left edge with a visible coiled spine, the pages slightly curved and smooth, and the cover showing a mild reflective sheen and soft glare over the image area, muting the colors and producing subtle highlights and shadows.
Original Video
EgoEdit result 1
LucyEdit result 2
✏️ Edit Instruction
Instruction: Transform the stainless-steel bowl with sprouts into a smooth, glossy white ceramic mug — simple curved handle, standard medium hand-held cylindrical shape — holding a light brown, milky liquid whose surface shows swirling motion with subtle ripples and reflective highlights from stirring.
Original Video
EgoEdit result 1
LucyEdit result 2
✏️ Edit Instruction
Instruction: Transform the black touchscreen smartphone with keyboard into a passport with a deep burgundy/maroon, leather-like cover that is smooth but slightly textured and shows minor edge wear; place a large gold-foil embossed crest dominating the center, add smaller gold lettering and a biometric chip symbol near the bottom—the foil should be highly reflective and produce bright specular highlights when illuminated, while the cover maintains a subtle matte sheen contrasting the shiny metallic emblem.
Original Video
EgoEdit result 1
LucyEdit result 2
✏️ Edit Instruction
Instruction: Transform the aida fabric with embroidered motif into an off-white bandana with a smooth, slightly satiny texture, folded into a neat square; one edge features a short woven fringe with a visible hem and a few loose threads, while gentle folds and soft shadows reveal a lightweight, drapey fabric with a subtle sheen
Original Video
EgoEdit result 1
LucyEdit result 2
✏️ Edit Instruction
Instruction: Transform the white aida square into a bone mask resembling an aged human skull with glowing cyan-blue eye sockets and jaw, emitting pale blue frost. Place this in a sunlit sewing nook with a vintage sewing machine and neatly stacked fabric bolts.
Original Video
EgoEdit result 1
LucyEdit result 2
✏️ Edit Instruction
Instruction: Transform the video into psychedelic poster art style
Original Video
EgoEdit result 1
LucyEdit result 2
✏️ Edit Instruction
Instruction: Change the patterned chart on the left to dark blue while keeping the embroidered cloth on the right unchanged.
Original Video
EgoEdit result 1
LucyEdit result 2
✏️ Edit Instruction
Instruction: Transform the small metal cup with brown powder into an glass mug with a clear, cylindrical body and a small loop handle; it is colorless but picks up silvery and faint greenish reflections from its surroundings, has a thick weighted base with concentric ridges, and a smooth glossy surface that creates strong specular highlights and subtle internal refractions that slightly distort background lines.
Original Video
EgoEdit result 1
LucyEdit result 2
✏️ Edit Instruction
Instruction: Turn the video into a depth map
Original Video
EgoEdit result 1
LucyEdit result 2
✏️ Edit Instruction
Instruction: Transform the white-front smartphone with brown case into an open brass pocket compass with its face transformed into a pool of glowing molten rock, the dial appearing to bubble and drip bright orange-red lava studded with ember-like sparks; the brass casing and lid showing a warm, polished gold sheen with a brushed-metal texture, while the molten surface gives off yellow-white highlights, flickering blue-white flame tongues, drifting smoke wisps and a shimmering heat-haze.
Original Video
EgoEdit result 1
LucyEdit result 2
✏️ Edit Instruction
Instruction: Transform the white-front smartphone with rose-gold case into a handheld calculator with a compact rectangular body and rounded corners, molded in light gray matte plastic with a slightly glossy, dark lcd display window and a small reddish-brown solar strip above it; the keypad features raised round and rectangular buttons in darker gray and black with white numerals and symbols, creating a two-tone contrast; the surface is smooth plastic with mild reflections on the display; there are no special visual effects beyond ambient highlights and soft shadows
Original Video
EgoEdit result 1
LucyEdit result 2
✏️ Edit Instruction
Instruction: Transform the pale-blue fabric scrap being trimmed into an unfolded white paper towel with a rectangular shape, visible fold lines, and a perforated scalloped edge; show a subtle embossed diamond-and-dot texture that gives a quilted appearance, matte white color with slight highlights where it curves, and soft creasing and shadowing when manipulated.
Original Video
EgoEdit result 1
LucyEdit result 2
✏️ Edit Instruction
Instruction: add knife in hands
Original Video
EgoEdit result 1
LucyEdit result 2
✏️ Edit Instruction
Instruction: Replace the white pen with green accents with a short handheld screwdriver: a bright red, rounded plastic handle with ribbed grip sections and a gray collar near the shaft; a slim metallic shaft extending from the collar to a narrow tip; the handle shows a glossy sheen with highlight reflections while the metal shaft has a muted, slightly brushed finish.
Original Video
EgoEdit result 1
LucyEdit result 2
✏️ Edit Instruction
Instruction: Change the lab bench to an electronics workbench with a soldering iron and parts trays.
Original Video
EgoEdit result 1
LucyEdit result 2
✏️ Edit Instruction
Instruction: add spoon in hands
Original Video
EgoEdit result 1
LucyEdit result 2
✏️ Edit Instruction
Instruction: add sponge in hands
Original Video
EgoEdit result 1
LucyEdit result 2
✏️ Edit Instruction
Instruction: Morph the turquoise gingham with white aida strip into a mask overgrown with phosphorescent fungi — a pale, cracked ceramic face half‑encased in moss and dense clusters of tiny mushroom stalks and caps; it emits a teal‑green/cyan glow from the eyes and fungal tips with purple accents, speckled bioluminescent spots and drifting spores, combining rough, organic mossy textures and delicate fungal stems with flaking porcelain and dark vein‑like cracks.
Original Video
EgoEdit result 1
LucyEdit result 2
✏️ Edit Instruction
Instruction: Apply holographic interface overlays
Original Video
EgoEdit result 1
LucyEdit result 2
✏️ Edit Instruction
Instruction: Morph the white aida fabric with candle motif into a leather coin pouch with a worn brown flap and brass snap whose front appears ruptured, letting a viscous, lava-like molten rock pour downward; the molten area glows bright yellow, orange and red with glossy highlights, bubbling embers and charred black edges on the leather, casting an intense inner light and a subtle heat-shimmer with drifting sparks.
Original Video
EgoEdit result 1
LucyEdit result 2
✏️ Edit Instruction
Instruction: Replace the white aida fabric with a carved oak folding mask featuring flame-like scrollwork and warm honey-brown tones. Change the background to a bright sewing nook with a pegboard of tools and colorful thread jars, and apply holographic interface overlays to the scene.
Original Video
EgoEdit result 1
LucyEdit result 2
✏️ Edit Instruction
Instruction: Turn the video into a canny edge map
Original Video
EgoEdit result 1
LucyEdit result 2
✏️ Edit Instruction
Instruction: Apply socialist realism poster style to the video
Original Video
EgoEdit result 1
LucyEdit result 2
✏️ Edit Instruction
Instruction: Morph the porous green pottery sponge into a leather pouch with a dense, moss-like overgrowth — the outer surface appears coarse and fibrous with a loose, netted texture and mottled olive-brown and dark-green tones; scattered across the moss are bright neon-green phosphorescent pinpricks and small glowing clusters that emit a soft, bioluminescent shimmer, giving the pouch a subtly pulsing, otherworldly radiance.
Original Video
EgoEdit result 1
LucyEdit result 2
✏️ Edit Instruction
Instruction: Turn the depth map into a video with the following description: the egocentric subject draws on a large white sketchpad resting on a wooden table. I hold a black fineliner pen between my fingers and trace bold outlines, filling paw prints and lettering "DOGS" while sketching dog silhouettes. My left hand, sleeved in green patterned fabric, steadies the paper near a white drawstring and a thin bracelet. Warm overhead light casts soft shadows; head-mounted camera looks down and follows my hand.

Website template adapted from EditVerse