The way you retrieve variables from Airflow can impact the performance of your DAGs Photo by Daniele Franchi on UnsplashWhat happens if multiple data pipelines need to interact with the same API endpoint? Would you really have to declare this endpoint in every pipeline? In case this endpoint changes in the near future, you will…

A few personal lessons learned from developing LLM applications Source DALL·E 3 prompted with “Operationalizing LLMs, watercolor”It’s been fun posting articles exploring new Large Language Model (LLM) techniques and libraries as they emerge, but most of the time has been spent behind the scenes working on the operationalization of LLM solutions. Many organizations are working…

When you think of AI, you might think of ChatGPT, AI-generated art, or maybe something like the Terminator. But let’s take a step back and ask the basic question, “What is AI?” AI is short for artificial intelligence — which may not tell us much because one of these words is problematic. The first word,…

What are they, where are they, and are they right for you? Photo by Hongwei FAN on UnsplashInput and output (I/O) operations refer to the transfer of data between a computer’s main memory and various peripherals. Storage peripherals such as HDDs and SSDs have particular performance characteristics in terms of latency, throughput, and rate which…

Run and evaluate monocular depth estimation models with Hugging Face and FiftyOne Monocular Depth heat maps generated with Marigold on NYU depth v2 images. Image courtesy of the author.Humans view the world through two eyes. One of the primary benefits of this binocular vision is the ability to perceive depth — how near or far…

To ensure you can follow along, we are using pandas 2.2.0, which is the latest version available at the time of writing this article. You are probably already familiar with performing aggregations in pandas using methods such as sum or min. You have also probably used these methods in combination with groupby. Therefore, it will…

1.1: What is Gradient Descent Image by DALL-E-2In machine learning , Gradient Descent is a star player. It’s an optimization algorithm used to minimize a function by iteratively moving towards the steepest descent as defined by the negative of the gradient. Like in the picture, imagine you’re at the top of a mountain, and your…

Discover the concepts and basic methods of causal machine learning applied in Python Photo by David Clode on UnsplashCausal inference has many tangible applications in a wide variety of scenarios, but in my experience, it is a subject that is rarely talked about among data scientists. In this article, we define causal inference and motivate…

Image by AuthorFunctions are essential in a data science project because they make the code more modular, reusable, readable, and testable. However, writing a messy function that tries to do too much can introduce maintenance hurdles and diminish the code’s readability. In the following code, the function impute_missing_values is long, messy, and tries to do…

Introduction to AutoGen and Mistral AI: AutoGen is a framework developed by Microsoft and designed to simplify the development of multi-agent applications, particularly in orchestrating LLM agents. Multi-agent applications involve systems where multiple LLM or multi-modal agents or entities interact with each other in the whole workflow to achieve specific goals or tasks. These agents…