This capstone project explores a multimodal AI application that combines conversational AI, image generation, image analysis, and semantic search. I built it to understand how these capabilities work together and to create a foundation for future experiments, such as retrieval-augmented generation (RAG) with text and images.
The application uses TypeScript with Next.js 16.3.4, React 19, Tailwind CSS 4, and LangChain.js. OpenRouter provides a single gateway to hosted models. Claude Haiku 4.5 handles chat and image analysis, while Gemini 2.5 Flash Image generates images. Uploaded images are stored in MinIO, with metadata in PostgreSQL and embeddings in ChromaDB. CLIP ViT-B/32 and OpenAI's text-embedding-3-small support image and text retrieval.
I evaluated two image-search approaches using 210 pictures and 30 written queries. CLIP achieved 76.7% rank-1 accuracy and 0.858 mean reciprocal rank (MRR), compared with 70.0% and 0.787 for keyword search based on image descriptions. I also measured a 6.7 percentage-point retrieval cost after standardizing privacy-preserving descriptions that excluded personal appearance details.
The project taught me to evaluate AI claims with experiments and to supervise code-generating models through explicit scope, approval, and verification. I documented limitations that testing alone could not reveal, including CLIP's difficulty with counting and English-only retrieval. Next, I plan to explore RAG-assisted multimodal chat and image-to-image generation, including more consistent comic creation.
