Mushroom hunting with LLMs: what can go wrong?

You see a mushroom in a forest, take a photo, and pass it to an AI chat.

It might help with identification… but should we trust it with our health or life?

Dataset

I was looking to create a dataset, but was happy to discover there is an existing FungiTastic dataset with 340k observations (615k photos) of 2.8k species, built by Czech computer-vision researchers, based on data from a citizen science project, Atlas of Danish Fungi. Labels are verified by experts, a part of them by DNA sequencing. Besides normal photos, it has masks, captions, microscopic and satellite images - much more than we need.

The dataset is designed for supervised training, with a classical train-test-val split. Rather than being random, it is by year: train up to 2021, validation 2022, test 2023. We don’t train on anything, so we use the test dataset. Using LLMs as they are is, in classical machine learning, called a 0-shot approach.

We care about normal photos one would take with a smartphone. To pick species that you could actually find in a forest (rather than in a lab or on a bathroom wall), I used lists of mushrooms in Poland, which has a strong foraging tradition. A popular pastime, mushroom hunting, has its own name, grzybobranie, where going into the woods with a knife is socially acceptable.

Speaking about safety, I picked two lists - one an official list of sellable mushrooms in Poland (47, of which 37 are in the dataset), and the other, the list of deadly mushroom species from Wikipedia, which, after intersection with this dataset, has 20 species. Interestingly enough, they intersect at Tricholoma equestre (en. yellow knight, pl. gąska zielonka), considered edible in some countries but deadly in others, with noted fatal incidents. The joy of working with real data full of contradictions!

Here are all 55 species. The photo is from the training set, so as not to leak the data. To see names, hover or click on a tile.

names in:EnglishPolishCzechUkrainianGermanLithuanianFinnishSwedishDanish

Boletus edulis edibleENpenny bunPLborowik szlachetnyImleria badia edibleENbay boletePLpodgrzybek brunatnyXerocomus subtomentosus edibleENsuede boletePLpodgrzybek zajączekXerocomellus chrysenteron edibleENred cracking boletePLpodgrzybek złotawyLeccinum scabrum edibleENbrown birch boletePLkoźlarz babkaLeccinum aurantiacum edibleENred-capped scaber stalkPLkoźlarz czerwonyLeccinellum pseudoscabrum edibleENhornbeam boletePLkoźlarz grabowySuillus luteus edibleENslippery jackPLmaślak zwyczajnySuillus granulatus edibleENweeping boletePLmaślak ziarnistySuillus bovinus edibleENJersey cow mushroomPLmaślak sitarzSuillus variegatus edibleENvelvet boletePLmaślak pstrySuillus grevillei edibleENlarch boletePLmaślak żółtyGyroporus castaneus edibleENchestnut boletePLpiaskowiec kasztanowatyGyroporus cyanescens edibleENbluing boletePLpiaskowiec modrzakCantharellus cibarius edibleENchanterellePLpieprznik jadalny (kurka)Craterellus cornucopioides edibleENhorn of plentyPLlejkowiec dętyHydnum repandum edibleENhedgehog mushroomPLkolczak obłączastyMacrolepiota procera edibleENparasol mushroomPLczubajka kaniaAgaricus campestris edibleENfield mushroomPLpieczarka polnaAgaricus bisporus edibleENbutton mushroomPLpieczarka dwuzarodnikowaAgaricus bitorquis edibleENpavement mushroomPLpieczarka miejskaAgaricus sylvaticus edibleENblushing wood mushroomPLpieczarka lśniącaAgaricus sylvicola edibleENwood mushroomPLpieczarka zaroślowaLactarius deliciosus edibleENsaffron milk capPLmleczaj rydzLactarius deterrimus edibleENfalse saffron milkcapPLmleczaj świerkowyLactifluus volemus edibleENfishy milkcapPLmleczaj smacznyArmillaria mellea edibleENhoney fungusPLopieńka miodowaKuehneromyces mutabilis edibleENsheathed woodtuftPLłuszczak zmiennyPleurotus ostreatus edibleENoyster mushroomPLboczniak ostrygowatyMarasmius oreades edibleENfairy ring champignonPLtwardzioszek przydrożnyCortinarius caperatus edibleENgypsy mushroomPLpłachetka kołpakowataClitocybe odora edibleENaniseed funnelPLlejkówka wonnaMorchella esculenta edibleENcommon morelPLsmardz jadalnyVerpa bohemica edibleENearly morelPLnaparstniczka czeskaTuber aestivum edibleENsummer trufflePLtrufla letniaAuricularia auricula-judae edibleENwood earPLuszak bzowyTricholoma equestre deadlyENyellow knightPLgąska zielonkaAmanita phalloides deadlyENdeath capPLmuchomor sromotnikowyAmanita virosa deadlyENdestroying angelPLmuchomor jadowityAmanita muscaria poisonousENfly agaricPLmuchomor czerwonyGalerina marginata deadlyENfuneral bellPLhełmówka jadowitaCortinarius rubellus deadlyENdeadly webcapPLzasłonak rudawyCalonarius splendens deadlyENsplendid webcapPLzasłonak wspaniałyLepiota brunneoincarnata deadlyENdeadly dapperlingPLczubajeczka cielistoczerwonawaLepiota subincarnata deadlyENfatal dapperlingPLczubajeczkaLepiota castanea deadlyENchestnut dapperlingPLczubajeczka kasztanowataClitocybe rivulosa deadlyENfool's funnelPLlejkówka jadowitaInosperma erubescens deadlyENdeadly fibrecapPLstrzępiak ceglastyEntoloma sinuatum poisonousENlivid entolomaPLdzwonkówka jadowitaHypholoma fasciculare poisonousENsulphur tuftPLmaślanka wiązkowaPaxillus involutus deadlyENbrown roll-rimPLkrowiak podwiniętyGyromitra esculenta deadlyENfalse morelPLpiestrzenica kasztanowataLactarius torminosus poisonousENwoolly milkcapPLmleczaj wełniankaConocybe rugosa deadlyENwrinkled conecapPLstożkówka pomarszczonaClaviceps purpurea deadlyENergotPLbuławinka czerwona (sporysz)

Benchmark

I asked models to:

Which mushroom species is it? Reply with a JSON array of the 5 most likely species as Latin binomials, most likely first. Output only the JSON array, nothing else.

I used 20 photos of each species from the test dataset, 1,040 in total. For a few rare species there were fewer, so I topped them up from the validation split. I used Latin names. Traditional names, while often vivid, are less standardized. Here are the results, both for the first guess and among the top five guesses. Code, the photo list and every model answer are in the repo, if you want to rerun it.

Gemini 3.6 and 3.7 Flash ace the chart, as in other visual benchmarks - yet here the lead is substantial. We already saw that Gemini 3.7 Flash is great at puzzle games. It is a bit of a surprise that they are not only cheap but also better at recognising mushrooms than the frontier Claude Fable 5 and GPT-5.6-Sol, lauded as the best OpenAI vision model. GLM-5.3-Flash absolutely aces at cost efficiency.

Mistakes

Let’s see what the mistakes are - to see if they are minor or deadly ones.

Most errors are harmless: a milkcap called another milkcap, a bolete called another bolete. The dangerous ones are not random. A deadly webcap is called a chanterelle - the same mistake that kills foragers. Here are a few examples.

Download as PNG

names in:EnglishLatinPolishCzechUkrainianGermanLithuanianFinnishSwedishDanish

46%correctdeath capAmanita phalloidesmuchomor sromotnikowymuchomůrka zelenáмухомор зеленийGrüner Knollenblätterpilzžalsvoji musmirėkavalakärpässienilömsk flugsvampgrøn fluesvamp deadly

17%mistaken for ediblegrisetteAmanita vaginatamuchomor mglejarkaAmanita vaginataAmanita vaginataAmanita vaginataAmanita vaginataAmanita vaginataAmanita vaginataAmanita vaginata, ochre brittlegillRussula ochroleucagołąbek bladyRussula ochroleucaRussula ochroleucaRussula ochroleucaRussula ochroleucaRussula ochroleucaRussula ochroleucaRussula ochroleuca

11%mistaken for inediblefalse death capAmanita citrinamuchomor cytrynowymuchomůrka citrónováмухомор цитриновийGelber Knollenblätterpilzgelsvoji musmirėkeltakärpässienivitgul flugsvampkugleknoldet fluesvamp

15%mistaken for poisonousdestroying angelAmanita virosamuchomor jadowitymuchomůrka jízliváмухомор білий смердючийKegelhütiger Knollenblätterpilzsmailiakepurė musmirėvalkokärpässienivit flugsvampsnehvid fluesvamp, eastern destroying angelAmanita bisporigeraAmanita bisporigeraAmanita bisporigeraAmanita bisporigeraAmanita bisporigeraAmanita bisporigeraAmanita bisporigeraAmanita bisporigeraAmanita bisporigera

39%correctfuneral bellGalerina marginatahełmówka jadowitačepičatka jednobarváгалерина оторочкуватаGift-Häublingeglinė kūgiabudėmyrkkynääpikkägifthättingrandbæltet hjelmhat deadly

23%mistaken for ediblesheathed woodtuftKuehneromyces mutabilisłuskwiak zmiennyopeňka měnliváопеньок літнійGemeines Stockschwämmchenpaprastoji kelmiukėkoivunkantosieniföränderlig tofsskivlingforanderlig skælhat, velvet shankFlammulina velutipespłomiennica zimowaFlammulina velutipesFlammulina velutipesFlammulina velutipesFlammulina velutipesFlammulina velutipesFlammulina velutipesFlammulina velutipes

13%mistaken for inediblecommon rustgillGymnopilus penetransłysak plamistoblaszkowyplaménka nevonnáGymnopilus penetransGeflecktblättriger FlämmlingGymnopilus penetranskangaskarvaslakkifläckig bitterskivlingplettet flammehat, scurfy twigletTubaria furfuraceaTubaria furfuraceaTubaria furfuraceaTubaria furfuraceaTubaria furfuraceaTubaria furfuraceaTubaria furfuraceaTubaria furfuraceaTubaria furfuracea

10%mistaken for poisonousmoss bellGalerina hypnorumhełmówka mszarowačepičatka mechováGalerina hypnorumGalerina hypnorumGalerina hypnorumsammalnääpikkämosshättingmos-hjelmhat, sulphur tuftHypholoma fascicularemaślanka wiązkowatřepenitka svazčitáопеньок сірчано-жовтий несправжнійGrünblättriger Schwefelkopfpuokštinė kelmabudėkitkerälahokkasvavelgul slöjskivlingknippe-svovlhat

16%correctsplendid webcapCalonarius splendenszasłonak wspaniałypavučinec nádhernýCalonarius splendensSchöngelber KlumpfußCalonarius splendensCalonarius splendensCalonarius splendenssirene-slørhat deadly

27%mistaken for ediblechanterelleCantharellus cibariuspieprznik jadalnyliška obecnáлисичка справжняEchter Pfifferlingvalgomoji voveraitėkeltavahverokantarellalmindelig kantarel, American slippery jackSuillus americanusSuillus americanusSuillus americanusSuillus americanusSuillus americanusSuillus americanusSuillus americanusSuillus americanusSuillus americanus

15%mistaken for inediblesaffron webcapCortinarius croceuszasłonak szafranowyCortinarius croceusCortinarius croceusCortinarius croceusCortinarius croceusCortinarius croceusCortinarius croceusCortinarius croceus, false chanterelleHygrophoropsis aurantiacalisówka pomarańczowalištička pomerančováлисичка несправжняFalscher Pfifferlingvoveraitinė guotelėvalevahveronarrkantarellalmindelig orangekantarel

26%mistaken for poisonoussulphur knightTricholoma sulphureumgąska siarkowačirůvka sírožlutáTricholoma sulphureumGemeiner Schwefel-RitterlingTricholoma sulphureumrikkivalmuskasvavelmusseronsvovl-ridderhat, yellow knightTricholoma equestregąska zielonkačirůvka zelánkaрядовка зеленаGrünlingžalsvasis baltikaskangaskeltavalmuskariddarmusseronægte ridderhat

30%correctfool's funnelClitocybe rivulosalejkówka jadowitastrmělka odbarvenáклітоцибе червонуватий отруйнийRinnigbereifter TrichterlingClitocybe rivulosaClitocybe rivulosagifttrattskivlingeng-tragthat deadly

49%mistaken for ediblethe millerClitopilus prunulusbruzdniczek największymechovka obecnáпідвишеньMehlräslingmiltagrybis vyšninisjauhosienimjölskivlinggråhvid melhat, field mushroomAgaricus campestrispieczarka polnapečárka polníпечериця звичайнаWiesen-Champignondirvinis pievagrybisnurmiherkkusieniängschampinjonmark-champignon

4%mistaken for inedibleLepiota ermineaLepiota ermineaLepiota ermineaLepiota ermineaLepiota ermineaLepiota ermineaLepiota ermineaLepiota ermineaLepiota ermineaLepiota erminea, fragrant funnelClitocybe fragransClitocybe fragransClitocybe fragransClitocybe fragransClitocybe fragransClitocybe fragransClitocybe fragransClitocybe fragransClitocybe fragrans

10%mistaken for poisonousclouded agaricClitocybe nebularislejkówka szarawaClitocybe nebularisClitocybe nebularisClitocybe nebularisClitocybe nebularisClitocybe nebularisClitocybe nebularisClitocybe nebularis, silky pinkgillEntoloma sericeumdzwonkówka jedwabistazávojenka hedvábnáEntoloma sericeumSeidiger RötlingEntoloma sericeumjauhorusokassilkesrödhättingsilkeglinsende rødblad

5%correctfatal dapperlingLepiota subincarnataczubajeczka różowawabedla namasověláлепіота рожеваLepiota subincarnataLepiota subincarnatailtaukonsieniLepiota subincarnatakødfarvet parasolhat deadly

32%mistaken for ediblewhite dapperlingLeucoagaricus leucothitespieczareczka różowoblaszkowaLeucoagaricus leucothitesLeucoagaricus leucothitesLeucoagaricus leucothitesLeucoagaricus leucothitesLeucoagaricus leucothitesLeucoagaricus leucothitesLeucoagaricus leucothites, fairy ring champignonMarasmius oreadestwardzioszek przydrożnyšpička obecnáопеньок луговийNelken-Schwindlinglauminis mažūnisnurminahikasnejlikbroskskivlingelledans-bruskhat

14%mistaken for inedibleearthy powdercapCystoderma amianthinumziarnówka ochrowożółtazrnivka osinkováCystoderma amianthinumAmiant-Körnchenschirmlingamiantinė šlakabudėkeltaryhäkäsCystoderma amianthinumokkergul grynhat, spotted toughshankRhodocollybia maculatamonetnica plamistaRhodocollybia maculataRhodocollybia maculataRhodocollybia maculataRhodocollybia maculataRhodocollybia maculataRhodocollybia maculataRhodocollybia maculata

35%mistaken for poisonousstinking dapperlingLepiota cristataczubajeczka cuchnącabedla hřebenitáLepiota cristataStink-Schirmlingdvokiančioji žvynabudėlėpuistoukonsienisyrlig fjällskivlingstinkende parasolhat, shield dapperlingLepiota clypeolariaczubajeczka tarczowataLepiota clypeolariaLepiota clypeolariaLepiota clypeolariaLepiota clypeolariaLepiota clypeolariaLepiota clypeolariaLepiota clypeolaria

9%correcthornbeam boleteLeccinellum pseudoscabrumkoźlarz grabowykozák habrovýпідберезник грабовийHainbuchen-RöhrlingLeccinellum pseudoscabrumpähkinäntattihasselsoppavnbøg-skælrørhat edible

73%mistaken for ediblebrown birch boleteLeccinum scabrumkoźlarz babkakozák březovýпідберезникGemeiner Birkenpilzlepšėlehmäntattibjörksoppbrun skælrørhat, penny bunBoletus edulisborowik szlachetnyhřib smrkovýбілий грибGemeiner Steinpilztikrinis baravykasherkkutattikarljohanssvampspiselig rørhat

7%mistaken for inediblebitter boleteTylopilus felleusgoryczak żółciowyhřib žlučníkгірчакGemeiner Gallenröhrlingaitrusis pušynbaravykissappitattigallsoppgalderørhat, false death capAmanita citrinamuchomor cytrynowymuchomůrka citrónováмухомор цитриновийGelber Knollenblätterpilzgelsvoji musmirėkeltakärpässienivitgul flugsvampkugleknoldet fluesvamp

3%mistaken for poisonouspoison pieHebeloma crustuliniformewłośnianka rosistaHebeloma crustuliniformeHebeloma crustuliniformeHebeloma crustuliniformeHebeloma crustuliniformeHebeloma crustuliniformeHebeloma crustuliniformeHebeloma crustuliniforme, stinking dapperlingLepiota cristataczubajeczka cuchnącabedla hřebenitáLepiota cristataStink-Schirmlingdvokiančioji žvynabudėlėpuistoukonsienisyrlig fjällskivlingstinkende parasolhat

29%correctfalse saffron milkcapLactarius deterrimusmleczaj świerkowyryzec smrkovýрижик ялиновийFichten-Reizkereglyninė rudmėsėkuusenleppärouskublodriskagran-mælkehat edible

48%mistaken for ediblesaffron milk capLactarius deliciosusmleczaj rydzryzec pravýрижик смачнийEdel-Reizkerpušyninė rudmėsėmännynleppärouskutallblodriskavelsmagende mælkehat, chanterelleCantharellus cibariuspieprznik jadalnyliška obecnáлисичка справжняEchter Pfifferlingvalgomoji voveraitėkeltavahverokantarellalmindelig kantarel

10%mistaken for inediblefalse chanterelleHygrophoropsis aurantiacalisówka pomarańczowalištička pomerančováлисичка несправжняFalscher Pfifferlingvoveraitinė guotelėvalevahveronarrkantarellalmindelig orangekantarel, rufous milkcapLactarius rufusmleczaj rudyLactarius rufusLactarius rufusLactarius rufusLactarius rufusLactarius rufusLactarius rufusLactarius rufus

3%mistaken for poisonouswoolly milkcapLactarius torminosusmleczaj wełniankaryzec kravskýвовнянкаBirken-Milchlingpiengrybis paberžiskarvarouskuskäggriskaskægget mælkehat, brown roll-rimPaxillus involutuskrowiak podwiniętyčechratka podvinutáсвинуха тонкаKahler Kremplingpilkoji meškutėpulkkosienipluggskivlingalmindelig netbladhat

82%correctparasol mushroomMacrolepiota proceraczubajka kaniabedla vysokáгриб-зонтик великийGemeiner Riesenschirmlingskėtinė žvynabudėukonsienistolt fjällskivlingstor kæmpeparasolhat edible

10%mistaken for edibleshaggy parasolChlorophyllum rhacodesczubajka czerwieniejącabedla červenajícíгриб-зонтик червоніючийGemeiner Safranschirmlingšiurkščioji žvynabudėpuistoakansienirodnande fjällskivlingægte rabarberhat, giant puffballCalvatia giganteapurchawica olbrzymiaCalvatia giganteaCalvatia giganteaCalvatia giganteaCalvatia giganteaCalvatia giganteaCalvatia giganteaCalvatia gigantea

1%mistaken for inedibleAmanita thiersiiAmanita thiersiiAmanita thiersiiAmanita thiersiiAmanita thiersiiAmanita thiersiiAmanita thiersiiAmanita thiersiiAmanita thiersiiAmanita thiersii, European white eggAmanita ovoideamuchomor jajowatyAmanita ovoideaAmanita ovoideaAmanita ovoideaAmanita ovoideaAmanita ovoideaAmanita ovoideaAmanita ovoidea

5%mistaken for poisonousgreen-spored parasolChlorophyllum molybditesczubajka zielonawaChlorophyllum molybditesChlorophyllum molybditesGrünsporiger RiesenschirmlingChlorophyllum molybditesChlorophyllum molybditesChlorophyllum molybditesgrønsporet rabarberhat, freckled dapperlingLepiota asperajeżoskórka ostrołuskowaLepiota asperaLepiota asperaLepiota asperaLepiota asperaLepiota asperaLepiota asperaLepiota aspera

It is interesting to see error rates by model. Here we focus on the conditional probability of an edible mushroom being classified as poisonous (false negative, taking edible as the positive class) and the much more dangerous probability of a poisonous mushroom being classified as edible (false positive).

Well, Muse Spark 1.2 looks safe only because it declines to guess so often. But even the top models, Gemini 3.6 & 3.7 Flash, have a dangerous error rate of around 12%. I love Qwen3.8 27B and it is great at local coding, but I wouldn’t trust it with mushroom safety: over a third of poisonous mushrooms are classified as edible species!

Caveats

Let’s not take for granted that people can always distinguish mushrooms. A single photo might not be enough either, even for an expert. For example, Fungi of the California Floristic Province at iNaturalist instructs:

You should take a minimum of THREE photos for each species that you document - in situ (preferably with habitat or substrate visible), a detailed shot of the pileus (cap), and a detailed shot of the stipe (stem) and lamellae (gills).

There is a reason why the Atlas of Danish Fungi relies on expert verification and DNA sequencing, not on photos alone.

At the same time, a common error among foreign mushroom hunters: a mushroom looks like an edible one from home, and they do not know the local deadly lookalike. I also tried to add the hint “a photo from Denmark” to see if it made the results better, but in this case it was not helpful for LLMs.

Conclusion

So, do not eat a mushroom because AI told you it is safe. That was true years ago, and it still is. AI slop in code, text, or design might be annoying. But it is fixable with a few prompts, or a manual edit. It is much better than having to prompt “Do I need a liver transplant?”

Vibe-mushroom-eating is not there yet! We don’t want to end up like your uncle and ChatGPT, from a lovely AI-generated short warning about AI hallucinations.

At the same time, if we want to learn rather than risk, Gemini 3.7 Flash is gold. I vibe-coded an app for myself to memorize local plants on top of it. I want to use it to learn the foliage around me.

And have you ever used AI to identify mushrooms?