S M Rayeed

S M Rayeed

I study vision-language models through visual question answering: what they see, what they retrieve, and when they should answer, in biological and biomedical domains.

PhD student in Computer Science
Rensselaer Polytechnic Institute, DAMI Lab

CV Email Scholar GitHub LinkedIn ORCID

Open to opportunities

Research internships and collaborations

I am open to research internships, industry collaborations and research-focused roles in multimodal AI, vision-language models and retrieval-augmented generation. If my work fits what your team is building, I would be glad to hear from you.

Get in touch
Q: What color is its crown?bird 0.98crownA: blue
Visual question answeringGround the question in the image, then answer
Q: What is the surface area of this lake?entity: Ełk LakeKnowledge base: Ełk LakeArea: 3.82 km²A: 3.82 km²
Knowledge-based VQARecognize the entity, retrieve, answer
Q: Segment the body partsheadpronotumelytralegsantennaeMask2Former, 85.1 mIoU
Fine-grained recognitionSegment parts of a specimen to measure traits
Q: Which disease does lovastatin treat?Retrieved page“Lovastatin is a statinprescribed to treathyperlipidemia.”names only the familyhyperlipidemiafamilialcombinedLPL-related+ 5 moredisease ontologyA: familial hyperlipidemia
RAG with partial evidenceThe page names only the disease family; an ontology narrows it

Hover over or tap a panel to run it.

S M Rayeed

About

  • Vision-language models
  • Visual question answering
  • Knowledge-based VQA
  • Retrieval-augmented generation
  • Knowledge graphs
  • Biological imaging

I am a PhD student in Computer Science at Rensselaer Polytechnic Institute, advised by Yao Ma and Charles V. Stewart in the Data Analytics and Machine Intelligence (DAMI) Lab.

I work on vision-language models and visual question answering in biology and biomedicine, two domains where fine-grained detail and checkable facts show exactly where models fail. I built Bio-VQA (WACV 2027), the largest biological VQA benchmark, with 1.86M questions across 18 VLMs, and co-authored an audit and repair of knowledge-based VQA benchmarks (ECCV 2026).

My current work is retrieval-augmented and agentic QA over knowledge graphs: what a model should do when a retrieved page names only part of the answer, and why biomedical text and the models trained on it under-report drug safety facts. Earlier, I benchmarked vision encoders for fine-grained recognition and built the detection pipelines and validated image datasets behind this work, including one in Nature Scientific Data.

News

  1. Bio-VQA is accepted to WACV 2027. The benchmark is public on Hugging Face, with Bio-VQA-Lite for low-cost evaluation.
  2. Our continental-scale ground beetle dataset is accepted at Nature Scientific Data. Hawaii-beetles and 2018-NEON-beetles are public.
  3. Our audit and repair of knowledge-based VQA benchmarks is accepted to ECCV 2026.
  4. Year 2 of the NSF HDR Scientific Modeling Out-of-Distribution ML Challenge, which I co-organized as a student organizer, closes with awards at the FARR Workshop.
  5. Two papers at the NeurIPS 2025 Imageomics workshop: BeetleFlow and Mind the (Data) Gap.
  6. Presented our fine-grained recognition benchmark at the ICCV 2025 CV4E workshop.
  7. The ML Challenge launches; its Imageomics track asks models to predict drought from NEON ground beetle images.
Earlier news
  1. Attended FuncaPalooza 2025, where BeetleFlow started.
  2. Two papers at the CVPR 2025 CV4Animals workshop: BeetleVerse and multiview track clustering for re-identification.
  3. Attended Wild Data: AI for Ecology and the Imageomics All-Hands Meeting.
  4. Attended BeetlePalooza 2024, where the ground beetle imaging effort began.

Research

All research areas
Multi-hop knowledge-based VQA over biotic interactions: identify the species in the image, follow a GloBI interaction edge to its partner, and answer from the partner's trait.

Knowledge-based VQA and retrieval

Retrieval-augmented QA that is measured correctly and knows what to do when its evidence is only partial, from open-domain KB-VQA to knowledge graphs.

3 papers, work in progressExplore this area
Fine-grained attribute grounding: naming, relative salience, morphological traits and visually prompted anatomical localization.

Visual question answering for biology

How finely do VLMs see living things? A 1.86M-question benchmark of classification, grounding, counting, measurement and identity.

1 paper

1.86M questions217K images18 open VLMs11 datasets

General visual skills do not transfer to biology: accuracy falls from .72 at phylum to .12 at scientific name, and most models measure traits no better than an image-blind baseline.

Explore this area
Imaging and trait-measurement workflow for pinned (top) and vial (bottom) specimens.

Vision datasets and detection pipelines

Open-vocabulary detection, segmentation and VLM verification at scale, and the validated datasets they produce.

4 papers, 2 datasetsExplore this area

Selected publications

All 14 publications
ConferenceJournalWorkshopFirst author
  1. WACV 2027First author

    How Finely Can Vision-Language Models See Biology? Bio-VQA: A Benchmark for Taxonomic, Morphological, Spatial and Individual Reasoning

    S M Rayeed, Qian Ma, Charles V. Stewart, Yao Ma
    IEEE/CVF Winter Conference on Applications of Computer Vision (WACV 2027)
    Project pagePaperCode
  2. ECCV 2026

    Identifying and Resolving Pitfalls of Knowledge-Based VQA Benchmarks: Auditing, Repairing, and Augmenting

    Qian Ma, S M Rayeed, Charles V. Stewart, Qiong Wu, Yao Ma
    European Conference on Computer Vision (ECCV 2026)
    PaperarXivCode
  3. Scientific Data 2026First author

    A continental-scale dataset of ground beetles with high-resolution images and validated morphological trait measurements

    S M Rayeed, Mridul Khurana, Alyson East, Isadora E. Fluck, Elizabeth G. Campolongo, Samuel Stevens, Iuliia Zarubiieva, Scott C. Lowe, Michael W. Denslow, Evan D. Donoso, Jiaman Wu, Michelle Ramirez, Benjamin Baiser, Charles V. Stewart, Paula Mabee, Tanya Berger-Wolf, Anuj Karpatne, Hilmar Lapp, Robert P. Guralnick, Graham W. Taylor, Sydne Record
    Scientific Data (Nature Portfolio), Data Descriptor
    Project pagePaperDOIData: Hawaii-beetlesData: 2018-NEON-beetlesCode
  4. ICCV 2025 WorkshopFirst author

    Fine-Grained Beetle Taxonomy with Vision Models: A Benchmark on Long-Tailed and Domain-Adaptive Classification

    S M Rayeed, Alyson East, Samuel Stevens, Sydne Record, Charles V. Stewart
    ICCV 2025 Workshop on Computer Vision for Ecology (CV4E)
    Project pagePaperSupplement
  5. CVPR 2025 WorkshopFirst author

    BeetleVerse: A study on taxonomic classification of ground beetles

    S M Rayeed, Alyson East, Samuel Stevens, Sydne Record, Charles V. Stewart
    CVPR 2025 Workshop on CV4Animals
    PaperarXivPoster
  6. MEE 2025

    Optimizing Image Capture for Computer Vision-Powered Taxonomic Identification and Trait Recognition of Biodiversity Specimens

    Alyson East, Elizabeth G. Campolongo, Luke Meyers, S M Rayeed, Samuel Stevens, Iuliia Zarubiieva, Isadora E. Fluck, Jennifer C. Girón, Maximiliane Jousse, Scott Lowe, Kayla I. Perry, Isabelle Betancourt, Noah Charney, Evan Donoso, Nathan Fox, Kim J. Landsbergen, Ekaterina Nepovinnykh, Michelle Ramirez, Parkash Singh, Khum Thapa-Magar, Matthew Thompson, Evan Waite, Tanya Berger-Wolf, Hilmar Lapp, Paula Mabee, Charles V. Stewart, Graham W. Taylor, Sydne Record
    Methods in Ecology and Evolution, 2025
    PaperarXiv

Datasets

All datasets

Background

Full CV (PDF)

Education

  1. Aug 2023 to presentPhD, Computer ScienceRensselaer Polytechnic Institute
    Advisors: Yao Ma and Charles V. Stewart. CGPA 3.96/4.00
  2. Aug 2023 to May 2025MS, Computer ScienceRensselaer Polytechnic Institute
    Project: Fine-grained beetle taxonomy with vision models. CGPA 4.00/4.00
  3. Jan 2017 to Mar 2021BSc, Computer Science and EngineeringIslamic University of Technology
    CGPA 3.76/4.00

Teaching

  1. Fall 2024CSCI 4965/6965 AI for ConservationInstructor: Charles V. Stewart
  2. Fall 2023 to Fall 2024CSCI 1100 Computer Science IInstructor: Uzma Mushtaque

Experience

  1. Jan 2024 to presentGraduate Research AssistantData Analytics and Machine Intelligence (DAMI) Lab, RPI
    Vision-language models, VQA and retrieval-augmented generation. Funded by the NSF Imageomics Institute (OAC-2118240).
  2. Apr 2021 to Jul 2023Research AssociateSystems and Software Lab, IUT
    Bangla sign language recognition. Funded by UIU-IAR-01-2022-SE-37.
  3. Mar 2021 to Jul 2023Software Quality Assurance EngineerNilavo Technologies (offshore facility of Biscom Inc.)
    Test automation with Playwright, Cypress and Selenium, API and security testing.

Service

  • Reviewer, KDD AI for Science track (2026 cycle 2, 2027 cycle 1)
  • Sub-reviewer, WACV 2027 and ECCV 2026
  • Reviewer, CVPR CV4Animals workshop (2025, 2026) and NeurIPS Imageomics workshop (2025)
  • Reviewer, Communications Earth & Environment (Nature Portfolio) and PLOS ONE (2026)
  • Student organizer, Scientific Modeling Out-of-Distribution ML Challenge (2026), by Imageomics, A3D3 and iHARP

Tools I work with

VLMs
Qwen3-VL, InternVL3.5, Gemma 3/4, LLaVA-NeXT, Llama 3.2 Vision, Pixtral, Molmo2, Phi-4-multimodal, Florence-2, BLIP-2
Encoders
CLIP, SigLIP, EVA-CLIP, BioCLIP, BioCLIP-2, DINOv2, DINOv3, ViT, ConvNeXt, Swin V2
Retrieval and VQA
Multimodal RAG, dense retrieval, re-ranking, knowledge-based VQA, knowledge graph QA, multi-hop QA
Detection and segmentation
Grounding DINO, YOLOv8 to v10, SAM, SAM2, MegaDetector, Detectron2, Moondream
HPC and inference
SLURM array pipelines on A100/H100 clusters, FlashAttention, FP8 quantization, HF Transformers