KDD 2027In preparationFirst author

BioHop: Multi-Hop Knowledge-Based Visual Question Answering over Species Interactions and Traits

S M Rayeed, Charles V. Stewart, Yao Ma

In preparation for the KDD 2027 Datasets and Benchmarks Track

A BioHop question shows only the photo of species A. The model must identify A, follow a GloBI interaction to partner B, which is never shown or named, and read B's trait from TraitBank. Photos from TreeOfLife-200M.
A BioHop question shows only the photo of species A. The model must identify A, follow a GloBI interaction to partner B, which is never shown or named, and read B's trait from TraitBank. Photos from TreeOfLife-200M.

BioHop shows a photo of one species and asks about a trait of its ecological partner, which is never shown and never named. Open VLMs answer barely above a wording baseline; supplying the partner closes 81 to 98% of the gap to the one-hop ceiling, so partner resolution is the bottleneck.

Knowledge-based VQA benchmarks such as E-VQA and InfoSeek ask about the entity in the image. In biology, much of what matters about an organism sits one hop away: what it eats, what hunts it, which plants it visits, and the traits of those partners.

Answering a BioHop question takes three steps: identify species A from the photo, follow a GloBI interaction to partner B, and read B's trait from TraitBank. Controlled forms of every question isolate each step, and each hop is checked against Wikipedia to see whether a text retriever could find the bridge at all.

How the knowledge graph is built

GBIF backbone keys are the hub. Wikidata maps each GBIF key to the EOL, NCBI and Open Tree of Life ids that GloBI and TraitBank use. The join yields one node per species with images and the paths that questions come from: species A interacts with species B, and B has a trait value.

  1. TreeOfLife-200M538,972 species, all with images
  2. GBIF backboneNames matched to accepted taxon keys, synonyms followed
  3. WikidataGBIF key to EOL, NCBI and Open Tree ids, plus the English Wikipedia article
  4. GloBI1,696,463 interaction edges among 113,545 species
  5. TraitBank1,107,365 trait values for 318,503 species, 58 predicates

Benchmark at a glance

Graph538,972 species; 1,696,463 GloBI edges, 900,280 (53%) gold, matched by id or by two or more records; 1,107,365 TraitBank values
InteractionsEats 655,847; preys on 464,496; has host 232,714; pollinates 166,661; visits flowers of 138,807; parasite of 37,938
Two-hop19,547 questions over 9,314 pictured species and 37 traits; 11,666 train, 2,051 validation, 5,830 test; 13,043 gold and 6,504 silver
One-hop controls70,872 questions about A itself, over 36,402 species and 45 traits
Three-hop5,141 chains A to B to C over 755 species and 44 traits; 395 with photos evaluated
ModelsQwen3-VL-32B and 8B, Qwen2.5-VL-7B, InternVL3.5-8B, Gemma-3-12B, LLaVA-OneVision-7B, greedy decoding; a LoRA-tuned Qwen2.5-VL-7B

Construction rules

A question is kept only when A has at most 10 recorded partners under the interaction, all of them share the trait value (the same range for numeric traits), and each partner's TraitBank records agree at least 80% of the time, so every answer is unanimous.

Answers are rebalanced within wording groups, which cut a question-only baseline from 50.7% to 35.3%. Splits are by pictured species A, so no test species appears in training.

Controlled forms give the model the photo of A (the task), only A's name, only A's kind (“an insect”), the photo plus the true partner B, or the one-hop question about B.

Partner resolution is the bottleneck

Closed-book accuracy is 36.7 to 40.3%, against a wording baseline of 35.3% and chance at 32.1%. Giving the model the true partner lifts every model, to 44.7 to 59.0%, closing 81 to 98% of the gap to asking about B directly.

Accuracy by form on the diagnostic set (3,993 questions): photo of A (the task), A's name, A's kind only, the true partner given, and the one-hop question about B.
Accuracy by form on the diagnostic set (3,993 questions): photo of A (the task), A's name, A's kind only, the true partner given, and the one-hop question about B.
ModelPhoto of AName of AKind onlyPartner givenB by photoB by name
Qwen3-VL-32B40.337.634.659.053.859.6
Qwen3-VL-8B37.436.235.052.051.353.7
Qwen2.5-VL-7B37.337.736.048.848.250.9
Qwen2.5-VL-7B + LoRA45.541.338.458.560.159.8
InternVL3.5-8B36.736.535.247.544.747.7
Gemma-3-12B37.035.934.351.449.754.8
LLaVA-OV-7B37.737.136.844.746.046.1
Wording baseline35.335.335.335.335.335.3
Chance32.132.132.132.132.132.1

Accuracy (%) by form on the diagnostic set of 3,993 test questions with one partner, both species extant and both with photos.

ModelPhoto of AKind onlyOne partnerSeveral partners
Qwen3-VL-32B40.634.740.243.4
Qwen3-VL-8B38.636.438.241.3
Qwen2.5-VL-7B38.435.838.339.0
Qwen2.5-VL-7B + LoRA46.838.345.853.7
InternVL3.5-8B37.135.736.839.1
Gemma-3-12B38.234.737.841.2
LLaVA-OV-7B37.536.137.537.6
Wording baseline36.536.536.536.5
Chance31.731.731.731.7

Closed-book accuracy (%) on the full two-hop test split: 5,300 questions with a photo, 4,692 with one partner and 739 with several.

Where the chain breaks

  1. VLMs cannot name the species. They name species A correctly 2.0 to 13.6% of the time and its genus 6.3 to 29.9%. BioCLIP-2 retrieves A zero-shot at 55.4% top 1 and 70.9% top 5; general CLIP-style retrievers stay below 12%.
  2. The graph route works where names do not. Handing the model BioCLIP-2's name for A does not help, but following GloBI from BioCLIP-2's predictions reaches the true partner 56.2% of the time from the top 1 and 60.1% routed over the top 5, raising accuracy to 42.8 to 51.3%. With the partner's TraitBank value, models reach 74.9 to 76.6%.
  3. Text does not carry the link. A's Wikipedia article names B for 9.7% of questions, and for none of the preys-on edges. Three retrieved passages move accuracy by at most 1.2 points, while the GloBI route adds 5.1 to 11.0. When the article does name B, retrievers find it 91 to 93% of the time.
  4. Relatives fill some gaps. With B's own record removed, the majority of evidence from B's genus or family is right 52.4 to 53.4% of the time, and 63.7 to 64.1% when taken from the true B's relatives.
  5. Three hops stay hard. On 395 chains, accuracy from the photo is 34.4 to 37.7% for the base models. Giving both links lifts Qwen3-VL-32B to 52.4%.
  6. Training helps, most on unseen partners. LoRA (rank 16, one epoch, 16,000 questions) lifts Qwen2.5-VL-7B from 38.4 to 46.8% on the full test split. The gain is largest, +10.0 points, for partners never seen in training, so it is not memorization of partners.
Identifying species A: zero-shot image retrievers over the benchmark's species, and VLMs naming the species in an open answer.
Identifying species A: zero-shot image retrievers over the benchmark's species, and VLMs naming the species in an open answer.
Retrieval ladder: closed book, BioCLIP-2's name for A, the GloBI partner of BioCLIP-2's top-1 and top-5 predictions, the true partner, and the partner with its TraitBank value.
Retrieval ladder: closed book, BioCLIP-2's name for A, the GloBI partner of BioCLIP-2's top-1 and top-5 predictions, the true partner, and the partner with its TraitBank value.
Wikipedia rarely states the link: A's article names B in 9.7% of questions and never for preys-on edges. When the article does name B, text retrievers find it 91 to 93% of the time.
Wikipedia rarely states the link: A's article names B in 9.7% of questions and never for preys-on edges. When the article does name B, text retrievers find it 91 to 93% of the time.
Change over closed book: three retrieved Wikipedia passages move accuracy by at most 1.2 points, while the GloBI graph route adds 5.1 to 11.0 points.
Change over closed book: three retrieved Wikipedia passages move accuracy by at most 1.2 points, while the GloBI graph route adds 5.1 to 11.0 points.
Gap filling with B's own trait record removed: evidence from B's genus or family, read by the model or by majority vote, against the majority from the true B's relatives.
Gap filling with B's own trait record removed: evidence from B's genus or family, read by the model or by majority vote, against the majority from the true B's relatives.
Three-hop chains A to B to C (395 with photos): from the photo of A, with the first link, with both links, and asking about C directly.
Three-hop chains A to B to C (395 with photos): from the photo of A, with the first link, with both links, and asking about C directly.
LoRA fine-tuning gains on the full test split, split by whether training questions involved the same partner B. The largest gain, +10.0 points, is on partners never seen in training.
LoRA fine-tuning gains on the full test split, split by whether training questions involved the same partner B. The largest gain, +10.0 points, is on partners never seen in training.
Pooled accuracy per trait over six models: from the photo of A, with the partner given, and asking B directly, against chance.
Pooled accuracy per trait over six models: from the photo of A, with the partner given, and asking B directly, against chance.

Robustness checks

  • Restricted to questions where the model named A correctly, accuracy from the photo rises (31.6 to 54.4%) but stays well below the partner-given form (35.4 to 64.7%).
  • On 52 questions where A's scientific name contains B's genus, accuracy from A's name rises by 6 to 23 points, so these leaks are tracked separately.
  • Gold and silver questions give similar accuracy once gold is reweighted to silver's trait mix.

Related work

  1. AAAI 2027 WorkshopUnder reviewFirst author

    CLADE: Category-Level Answering from Deficient Evidence in Retrieval-Augmented Language Models

    S M Rayeed, Charles V. Stewart, Yao Ma
    arXiv preprint, 2026. Under review at the AAAI 2027 NeuSymBridge Workshop
  2. AAAI 2027 WorkshopUnder reviewFirst author

    LABEL: Literature Asymmetry between Efficacy and Limits in Drug Knowledge and Language Models

    S M Rayeed, Charles V. Stewart, Yao Ma
    arXiv preprint, 2026. Under review at the AAAI 2027 Workshop on Health Intelligence (W3PHIAI)
  3. WACV 2027First author

    How Finely Can Vision-Language Models See Biology? Bio-VQA: A Benchmark for Taxonomic, Morphological, Spatial and Individual Reasoning

    S M Rayeed, Qian Ma, Charles V. Stewart, Yao Ma
    IEEE/CVF Winter Conference on Applications of Computer Vision (WACV 2027)