With tariffs kicking in and imports slowing, I want to track the local impact at my grocery stores. Does anyone have any suggestions? Would scraping websites of local grocers be sufficient? Any prior art?
This can only be done with crowdsource data. The hard parts, besides accumulating active users, are unique product identification and reputation system to prevent fake data flooding.
Autoscraping the price tags, as the sibling suggested, would be nice. However, in real retail the codes on the tags are regularly as small as possible and sometimes do not even correspond with the one on the product (don't know how this works). Means you need to scan them really close, individually. You also can't rely on placement for augmenting missing/trimmed textual data on tags with identifying the actual item on the shelf as in many places it's consistently messed up (as if on purpose).
I wanted to build this a couple of years ago. My idea for solving the user problem was as follows: - Users get to ask whether they are getting the best prices (or not) for an item. - They do this by scanning the label with barcode. - If anything is unclear they are asked to confirm the item, price, store, etc... - Once confirmed they get to see price history or prices at other nearby stores. That way you get a new datapoint per request.
You could probably solve some of the data integrity problems with some data science.
Tools are better now than they were back then so perhaps you could vibe code this in a couple of hours?
These days I would like to pair this with a database of the supply chain of products which identifiers were parts are sourced from etc... I could then enter some preferences like "avoid products from countries engaging in genocide" and after scanning such a product it could tell me if a product failed one of my filters.
I did some research towards this end and found some existing tools that could be useful:
- https://github.com/Forceu/barcodebuddy/ (Barcode system for Grocy)
- https://world.openfoodfacts.org/ (Open Food Facts is a food products database made by everyone, for everyone. You can use it to make better food choices, and as it is open data, anyone can re-use it for any purpose.)
We tend to think that our only power is at the voting both and there it often feels negligible. I believe that we actually hold a lot of economic power through our consumption choices, especially those of us in first world economies or with higher income. Not only could this be more influential that casting a vote every four or five years, but it actually gives you the opportunity to express a much more nuanced view rather than picking one of two (or a handful of) politicians who represent a portfolio of policies that might not be that different from each other (thus offering no real choice) or where you have to pick the lesser of two evils (thus implicitly endorsing policies you disagree with). Currently this economic power is largely squandered (except for extreme cases like boycotts), like misaligned magnetic dipoles in a ferromagnet cancelling each other out in the absence of an external magnetic field. If instead we could get all the individuals to align, consumers in the economic force case or atomic or molecular magnetic dipoles in the magnetism case, then the same collection of individuals is able to produce a powerful force field with real effects.
Instead of parsing data on things on shelves with video for the data points why not just use receipts? That correlates more closely to the price data than the consistency of what's actually on the shelves, though video of the shelves would also be useful (i.e. estimating inventory/shortage).
I am not sure how the system for most grocery stores works now, but in the ancient past there was a central authority that set the prices for individual stores (so in Australia, the Coles or Woolworths all have individualised prices per store for items except for advertised sale items).
The computer in the store that drives the POS scanners then can print out the shelf label stickers and a poor sod goes and replaces the stickers on the shelves.
This is all a big secret of course, so asking a grocery store for their prices is going to get you nowhere.
You could try scraping the websites, but often the price for online shopping is different to what is in store.
Best bet would be to offer an incentive for people to scan their receipts into your app. I don't know what kind of incentive you could build to do that, maybe for every 5 receipts you could offer a coupon or something. Not much of an idea and a massive PITA, it would also only give prices on those things that people bought.
edit: also gets complicated for multi-item discounts or combination discounts
There are barcode scanning libraries that specialise in scanning dozens or hundreds of barcodes in a single frame.
These are typically used to mass-import your competitors prices just by walking through the shop with a camera mounted on your shoulder. Or phone in a shirt pocket with the camera facing forward.
There’s a few white-label companies that offer these apps. Many of the barcode companies offer their own version of this app. They’re not usually available on the AppStores.
The reason for this is that each competitor requires slight customisation (font, price location relative to each barcode, etc) to get the best results.
I thought barcodes were effectively just "guids" that you could use for a lookup table or whatever to get pricing etc. Knowing what the product _is_ wouldn't tell you what the price is - you'd need the table.
How does it actually work?
It seems to me the way to do this is have folks scan in a receipt. Tag with any missing information (location, date) that may be missing.
Incentivize the behavior... somehow... and hope for the best.
Safeway’s site is extremely aggressive with anti scraping. Even if you simply direct load a product url a single time you’re hit with a captcha. Try it - view any product page, and the change 1 character in the product id in the url. Instant captcha. Scrapers be damned.
You think they might be checking the clickstream/referrer header?
Flipp ? Just scrape Flipp ?
Your grocery store would have significant data that it feeds its e-ink displays with. Capturing those updates could be fun if you’re looking for a project…
many grocery stores here don't even publish prices in a machine readable format, or only for selected products. best of luck.