Glass Pantry Vision Lab
Checking…Using OpenCV to upgrade our fridge app.
This is a lab for Glass Pantry, the fridge app we are building (an early build, in private testing), not the app itself: we test vision here, and what passes goes into the app.
Photograph a shelf again, and OpenCV 5 shows what may have changed before any AI is asked. Every flag is a question: you confirm, and nothing is deleted.
Between these two photos one pack was taken out and one tub moved. OpenCV flagged 6 things as possibly changed: those 2, and 4 false alarms, each only a question to you. It cleared 7 of the 14 without an AI call.
A real pair from our fridge, run through this engine (whole photos). Each line joins a point OpenCV 5 found in both photos (ALIKED + LightGlue).
- Green: still there.
- Amber: not seen where it was, so you are asked.
- Blue dashed: looks new.
How it works
- Set up your fridge. Pick the model (or photograph its sticker) and trace each compartment on a photo; OpenCV straightens it.
- Photograph each shelf. Claude lists what is on it; OpenCV checks the photo and has small labels read again up close. You confirm, and the shelf appears in a 3D fridge.
- Photograph it again later. OpenCV lines the new photo up with the last one and checks each thing, with no AI. It marks possible changes and new spots; you may ask Claude to read just those. Things OpenCV finds only moved keep their reading. Nothing leaves the shelf unless you say so, and then it moves to Unplaced, never deleted. (Measured 10/26: one Claude call given both photos asked about far fewer unchanged things, for about a third of the cost of those close-up reads, so it is planned as step two.)
What we measured
- Photograph it again, 6 before/after pairs: every removal, the move and the addition were found, but it failed its bar on false alarms (8 false item flags; classic SIFT: 38). So it is marked experimental. 56 of 71 things were marked no visible change (SIFT: 27); with the code as shipped, 51 of 71, and 8 false new spots instead of 7. The page compares one shelf strip; that setup was not part of the test.
- Against the AI alone (10/26, 12 pairings of 12 photos from one kitchen we had used before, protocol frozen first): one Claude call given both photos and the old list was credited with every removal (5), move (2) and addition (5) in all three runs and asked about 27 to 29 of 148 unchanged things (OpenCV's first pass: 66), for about $0.03 and 4.7 s a pair. Our measured OpenCV-then-close-up stages took about $0.09 and 16 s a pair, so that single call is planned as step two. Among the passes with no AI call, OpenCV 5's learned matcher cleared the most unchanged things under shared rules: 82 of 148; the same ALIKED points without LightGlue cleared 29 and could not line up 5 of 6 re-paired photos; SIFT 33, ORB 20.
- Second looks at small labels: no wrong first reading to catch in 28; no accuracy gain claimed.
- Speed on AWS, OpenCV comparing one recorded pair (spot checks): one pair took 4.4 s, and 9.0 s on its first call. After we added a ping every 5 minutes to keep it warm (best effort), another pair took 5.1 s on its first call, then 4.8 s.
One kitchen and few photos: these show behaviour, not general accuracy.
Before you try it: where your photos go
- Photos are processed by this engine on AWS, in memory, without being intentionally kept; it logs only codes and counts. A photo or crop that Claude reads also goes to Anthropic. Tracing and comparing two photos use AWS only. Leave people and personal documents out of the frame.
- Your fridge and the last photo of each shelf are kept in this browser on this device, never on our server, so you can come back later. Anyone using this browser can see them; the fridge page has a button to forget them.
- Live reading is capped per day for everyone together; the recorded shelf check always works.