UHG
Search
Close this search box.

Python Can’t Get Package Management Right

Python cannot handle two different versions of the same package which leads to “dependency hell”, causing entire installations to fail.

Share

Python-Can’t-Get-The-Package-Management-Right
Listen to this story

The struggle is real. When a developer uses multiple package managers, there’s a risk of modules being overwritten or conflicting. Nearly 4.71% of packages on PyPI have module conflicts in their dependency graphs, leading to broken environments or even security risks.

Often, different package managers use different lock file formats, which can cause issues when switching between tools or collaborating with others using different package managers. In Python, things can get much worse when you consider dependency management.  

In the Python world there are multiple package managers including pip, Conda, Poetry, pipenv, pyenv which seem to have their own flaws. 

Why it matters? This makes it confusing for both new as well experienced developers, and eventually things feel unreasonably slow. Most users try to solve this by replicating other’s environment without giving it a second thought, and that does not work either.

But, Python is deaf for dependency resolution 

One of the primary issues in Python dependency management is handling conflicting dependencies

For instance, pip, the default package manager, cannot handle two different versions of the same package. This situation, often touted as “dependency hell”, causes entire installations to fail, leading to unexpected behaviour in projects.

A few months ago, one of the Reddit users mentioned that Python really feels like a programming environment from the early 80s, where a developer had a single project on their pc, and that was all they worked on for years. 

“Python wants to do everything on the global system level, including runtime versioning and packages. That means that any two developers can think they have a working project on their system, even though they have radically different setups. This makes handing off and deploying Python applications a nightmare,” he added further, suggesting why dependency resolution is a nightmare on Python. 

However, the most important and weird part of the dependency resolution is that pip makes assumptions. The pip documentation on dependency resolution explains that pip makes assumptions about package versions during installation and later checks these assumptions, which can lead to conflicts if the assumptions are incorrect.

Managing dependencies can be resource-heavy. One user reported having about 100+ GB of their hard drive filled with Python virtual dependencies, highlighting the storage impact of multiple environments.

Ergo, Virtual Environments

“I’m afraid of having 2000 folders, each one with a different virtual environment,” said one Reddit user expressing confusion about virtual environments. Running a project solely or in isolation becomes cumbersome. 

While virtual environments are essential for project isolation and dependency management, there are instances where users find virtual environments problematic rather than solving the problem. 

Previously, users have reported that package versions and dependencies can still conflict within virtual environments, requiring manual resolution in some cases that directly question the isolation in Python. 

Some developers view virtual environments as wasteful, believing they unnecessarily duplicate libraries for each project. As one Reddit user stated, “It seems like you’re installing a new copy of every library every time you start a new project, which seems like a waste of resources.”

The complexity of virtual environments can be overwhelming for those new to Python. A Reddit user expressed extreme frustration, saying, “I spend way more time just trying my computer to get my virtual environment up, project dependencies installed, and IDE configured than I do actually coding.”

Several developers recommend using Docker to avoid virtual environment issues altogether. This approach encapsulates the entire environment, making it more reproducible across different systems.

📣 Want to advertise in AIM? Book here

Picture of Sagar Sharma

Sagar Sharma

A software engineer who loves to experiment with new-gen AI. He also happens to love testing hardware and sometimes they crash. While reviving his crashed system, you can find him reading literature, manga, or watering plants.
Related Posts
Association of Data Scientists
Tailored Generative AI Training for Your Team
Upcoming Large format Conference
Sep 25-27, 2024 | 📍 Bangalore, India
Download the easiest way to
stay informed

Subscribe to The Belamy: Our Weekly Newsletter

Biggest AI stories, delivered to your inbox every week.

Flagship Events

Rising 2024 | DE&I in Tech Summit
April 4 and 5, 2024 | 📍 Hilton Convention Center, Manyata Tech Park, Bangalore
Data Engineering Summit 2024
May 30 and 31, 2024 | 📍 Bangalore, India
MachineCon USA 2024
26 July 2024 | 583 Park Avenue, New York
MachineCon GCC Summit 2024
June 28 2024 | 📍Bangalore, India
Cypher USA 2024
Nov 21-22 2024 | 📍Santa Clara Convention Center, California, USA
Cypher India 2024
September 25-27, 2024 | 📍Bangalore, India
discord icon
AI Forum for India
Our Discord Community for AI Ecosystem, In collaboration with NVIDIA.