ask.py: Perplexity like search-extract-summarize flow in Python

I’ve been experimenting with ask.py. to create a search-extract-summarize flow — something similar to what AI search engines like Perplexity offer.

While it’s a simplified version compared to what’s out there commercially, it’s a great way to get hands-on experience with the core concepts.

ask.py Workflow

Given a query, here’s the step-by-step process the program follows:

1. Web Search: It starts by querying Google to fetch the top 10 relevant web pages.
2. Content Scraping: It then crawls each of these pages to extract their textual content.
3. Chunking and Storage: The extracted text is broken down into smaller chunks and stored in a vector database.
4. Vector Similarity Search: Using the original query, it performs a vector search to find the top 10 matching text chunks.
5. [Optional] Hybrid Search: It can also perform a full-text search and combine those results with the vector search.
6. [Optional] Re-ranking: A reranker can be used to refine the ordering of the top chunks.
7. Answer Generation: The selected chunks serve as context for an LLM (Language Model) to generate a coherent answer.
8. Output with References: Finally, it outputs the generated answer along with references to the source material.