For any questions, comments, or feedback, please send them to:
- Stuti Pandey at stuti.pandey-1@colorado.edu
- Josh Myers Dean at Josh.Myers-dean@colorado.edu
- Jarek Reynolds at Jarek.Reynolds@colorado.edu
Lateral flow tests (LFTs) enable rapid, low-cost testing for health conditions including Covid, pregnancy, HIV, and malaria. Automated readers of LFT results can yield many benefits including empowering blind people to independently learn about their health and accelerating data entry for large-scale monitoring (e.g., for pandemics such as Covid) by using only a single photograph per LFT test. Accordingly, we explore the abilities of modern foundation vision language models (VLMs) in interpreting such tests. To enable this analysis, we first create a new labeled dataset with hierarchical segmentations of each LFT test and its nested test result window alongside a textual description of each image. We call this dataset LFT-Grounding. Next, webenchmark eight modern VLMs in zero-shot settings for analyzing these images. We demonstrate that current VLMs frequently fail to correctly identify the type of LFT test, interpret the test results, locate the nested result window of the LFT tests, and recognize LFT tests when they partially obfuscated. To facilitate community-wide progress towards automated LFT reading, we publicly release our dataset.
We provide JSON files with annotations for both segmentation masks and bounding boxes showing the location of each lateral flow test and its result window.
Each segmentation mask is described using the following format:
{
"Covid-Test-Positive-62P.jpg": {
"Total Polygons": 2,
"Ground Truth Dimensions": [
972,
1920
],
"Positive": true,
"Four Corners": [
[
0,
0
],
[
864,
0
],
[
864,
1920
],
[
0,
1920
]
],
"Covid Test": [
[
[
563,
1480
],
[
165,
720
],
[
179,
680
],
[
259,
620
],
[
338,
594
],
[
415,
606
],
[
795,
1316
],
[
799,
1356
],
[
750,
1416
],
[
691,
1464
],
[
634,
1490
],
[
591,
1494
]
]
],
"Test Result": [
[
[
452,
1020
],
[
489,
1002
],
[
597,
1202
],
[
555,
1222
],
[
455,
1028
]
]
],
"URL": "https://bivpriv.colorado.edu/covid_lft/Covid-Test-Positive-62P.jpg"
},
}Each bounding box is described using the following format:
{
"Covid-Test-Positive-62P.jpg": {
"Four Corners": [
0,
0,
864,
1920
],
"Covid Test": [
165,
594,
799,
1494
],
"Test Result": [
452,
1002,
597,
1222
],
"URL": "https://bivpriv.colorado.edu/covid_lft/Covid-Test-Positive-62P.jpg"
},
}We also provide code to demonstrate how to load, resize, generate, and save [binary masks] and [bounding boxes] for all images in the dataset.
BibTex Code HereFor any questions, comments, or feedback, please send them to: