Why a funnel is the centre of the war between data’s heaviest hitters Unstructured data takes varying forms. It’s typically text-heavy, but may contain data such as dates, numbers, and dictionaries as well. Data Engineers commonly encounter unstructured data in the form of deeply-nested jsons. However the term “unstructured” data really refers to anything non-tabular;…

You might say 2023 was an eventful year for data scientists and ML professionals, but that wouldn’t quite capture the amount of hectic activity we’ve seen in the field in the past 12 months. As much as we always aim to resist hype and hyperbole, we have to concede that yes, we’ve seen some dramatic…

ChatGPT, Generative AI A focus on configuring the temperature, the Top P, the frequency penalty, and the presence penalty directly in your ChatGPT prompts Photo by Growtika on UnsplashI’ve recently been reading a very interesting book by David Clinton, entitled The Complete Obsolete Guide to Generative AI, published by Manning Publications. In the second chapter,…

Delving into one of the most common nightmares for data scientists Introduction One of the biggest problems in linear regression is autocorrelated residuals. In this context, this article revisits linear regression, delves into the Cochrane–Orcutt procedure as a way to solve this problem, and explores a real-world application in fMRI brain activation analysis. Photo by…

RAGAs (Retrieval-Augmented Generation Assessment) is a framework (GitHub, Docs) that provides you with the necessary ingredients to help you evaluate your RAG pipeline on a component level. Evaluation Data What’s interesting about RAGAs is that it started out as a framework for “reference-free” evaluation [1]. That means, instead of having to rely on human-annotated ground…

Demystifying the confusion about Linear Regression Visually and Analytically Image: Linear regression illustration, by Stpasha, via Wikimedia Commons (Public Domain). Original Image Link: https://upload.wikimedia.org/wikipedia/commons/8/87/OLS_geometric_interpretation.svgThe above image represents a geometric interpretation of Ordinary Least Squares (OLS) or Linear Regression (words used interchangeably in classical statistics). Let’s break down what we’re seeing in an intuitive way: Variables…

Using reinforcement learning from scratch to teach a computer to play Tic-Tac-Toe Image by author (created with ChatGPT)It appears that everyone in the AI sector is currently honing their Reinforcement Learning (RL) skills, especially in Q-learning, following the recent rumours about OpenAI’s new AI model, Q* and I’m joining in too. However, rather than speculating…

A guide for developers and architects especially those who are not specialized in machine learning to design an MLOps architecture for their organization Introduction According to Gartner’s findings, only 53% of machine learning (ML) projects progress from proof of concept (POC) to production. Often there is a misalignment between the strategic objectives of the company…

The dos and don’ts of processing data in the age of AI 10 min read · 23 hours ago Image by Gilles Lambert on UnsplashThe digital economy has been built on the wonderful promise of equal, fast, and free access to knowledge and information. It has been a long time since then.…

Where the assumptions behind the two-tower model architecture break — and how to go beyond (Image created by the author using generative AI)Two-tower models are among the most common architectural design choices in modern recommender systems — the key idea is to have one tower that learns relevance, and a second, shallow, tower that learns…