note: slide cues are the lines that begin with ### ### Why are you here? Thank you all for coming to my talk. I'll assume you're here because you've either experienced cpan hell and want to learn how to fix it - or you're just early to get in the front row for miyagawa's talk. :) ### [miyagawa] ### Why are you here? I have to admit up front, as I started working from idea to abstract to presentation, I found that there's a lot more to this whole thing than I initially thought. ### What is CPAN Dependency Hell? The main reason for that is that there are a few different definitions for CPAN dependency hell. So... What is CPAN Dependency Hell? [...ask the audience] Probably the most common issue I've heard about ### Difficulty installing stuff are people who have had considerable difficulty getting things installed from CPAN. For example, difficulty upgrading their CPAN.pm. This talk isn't really about solving that but I can touch upon some tips later. ### Broken distributions Another definition of cpan dependency hell comes from module distributions that are just plain broken - and the modules you need RIGHT NOW depend on that one. I think I have some good advice for you, stay tuned. ### How I ended up doing this talk However, the idea for this talk came about because of a particular problem that my employer faced, and I ended up finding a way to fix and manage it using some of the excellent new tools that the perl community has developed in the last few years, namely git, local::lib, perlbrew, and also the old, trusty minicpan. #################################################################### ### [Pic of Dante] Now, like Dante Allegheri, let me describe my journey through CPAN Dependency Hell ### My descent into The Inferno In my particular case, I was working for a client whose main production web server was about six years old, still running CentOS 3, perl 5.8.0, and they had a number of problems that needed to be overcome. ### Everything was old First off, they wanted to use new modules but those wouldn't install on such an old perl. Second, they needed modules that depended on modules that wouldn't install on such an old perl. ### Everything was brittle THEN, they needed modules that depended on modules that if upgraded, were not backward compatible with their old versions and would cause the site to BREAK. ### $work was in deep By the way, they had been experimenting with starting from scratch and using Ruby to avoid these issues. ### Yeah, right Looks like we have some people here who have experienced Gems dependency hell. :) So, basically, they were in a bind. Luckily, because they were careful, they suffered few breakages of the production site, but were always fighting these issues on the staging box and their dev systems. Also luckily for them, somebody had the foresight to make the staging box a virtual machine and could roll-back to a known state when things broke... ### But they were still stuck but that's really not a sustainable solution, especially when everything you depend on is swiftly becoming deprecated and unsupportable. ### Abandon all hope, ye who enter here I personally had immense difficulty because I wanted to do my development with the latest perl and the latest CPAN modules and an operating system that hadn't yet passed it's expiration date. ### [pic of hell] We were in dependency hell, and needed a way out. #################################################################### ## How does one end up here? Now, I think I've just described a pretty common scenario: stuck on an old perl, stuck on an old OS, your app depends on old cpan modules, and you're practically paralyzed for fear of breaking it. Part of the reason so many people end up in this situation is because of several of perl's own strengths: ### Hoist by our own petard hoist by our own petard. perl is an integral part of just about every unix-like OS. installing modules from the cpan is ridiculously easy (most of the time) when you use the most common means of adding/upgrading modules, they're instantly available to your whole system. ### The CPAN is insanely convenient It's all so convenient that most of us simply don't think about it until it's TOO LATE. ### The CPAN is ever-changing The problem is that once you begin extending your perl environment using the standard means, you create a system that is nearly *impossible* to recreate in the future. The CPAN is ever-changing! Given an app with even a reasonably small set of dependencies on cpan modules, it is highly unlikely that you will be able to install the *exact* same versions of those modules in a month's time. ### The CPAN is NOT predictable Following the chains of dependencies, you may end up with additional modules, or missing ones. Perhaps a single module in your dependency chain has been upgraded and is no longer compatible with the version of perl you are using... and now you can't even install your app on a new system that is otherwise identical to the original. ### The CPAN is not made of magic ponies You believed that the CPAN was made of rainbows and magic ponies, and now you're stuck, frustrated, and worse yet - you still don't have a pony. ### [no, you can't have a pony - not yours] This is how you get to CPAN Dependency Hell. ### Welcome to Hell ### [flaming horns] #################################################################### ### But wait! But, look - we're NOT alone with this problem! Perl just has the easiest means of getting in deeper than most other languages. ### Perl is not alone Java, Ruby, Python, and PHP all have these issues... Ruby in particular suffers from it horribly - and I think part of that is because they have Gems which in some ways is similar to cpan but, in my opinion, still isn't nearly as awesome. ### Other languages punt Many apps in these languages bundle up all their dependencies in private lib dirs, with the versions of each third-party module and library carefully selected. In Java, Python, and Ruby apps, they will frequently require a specific version of their respective VM, and it's not uncommon for a Java app to bundle it in as part of the application! However, just because those languages have to do that doesn't mean perl does, too. ### Perl can be saved! We're better than that! As always, TIMTOWTDI... and as you may have suspected, the very thing that spoils us so much, the CPAN, is also what can save us! ### CPAN to the... rescue? As a side note, it also helps that the newest versions of Perl retain backwards compatibility pretty much to the point of insanity. This is already a hairy enough problem without having to worry about that. I can vouch that ruby, python, and java all make some things very painful because of non-backward compatibility. In the last few years, a number of tools have been created that, when combined creatively, will allow you to clean up the mess you're in and take control so you don't have to end up like that again. ### Take back control #################################################################### ### But how? Let me be your guide ### [virgil] ### Shiny new tools! ### [ List of tools ] * perlbrew * minicpan / CPAN::Mini * mcpani / CPAN::Mini::Inject * dpan / MyCPAN::App::DPAN * git * local::lib ### And many others * dpanneur * CPAN::Site * CPAN::Cache * CPAN::Mini::Webserver * CPAN::Unwind * Module::Depends * etc... ### How did I do it? First I started by trying to ### Track down dependencies track down all the dependencies of my client's codebase. I was able to automate a bunch of it, but it was still a roayl PITA. ### minicpan Then I created a minicpan from the latest & greatest from the CPAN... and tried to get a fully working site... manually. This involved finding things that failed to install or load or work, then tracking down a previous version that might work on the BackPAN, injecting that into the minicpan (replacing the newest version) then trying to make that work, ### lather, rinse, repeat and lather, rinse, repeat, ad nauseam. ### That was the hard way Yes, I did it the hard way, and it sucked and took weeks. ### [ shaved yak ] In the process I half-shaved a lot of yaks ### My $client is a saint... Because of the extra hours burned doing that, my client is officially a saint. ### Please use them to sell, trade or rent a timeshare ### Using the tools But anlong the way I learned a whole lot about the ecosystem that we call the CPAN. I also started to pull together the concepts that became this talk. ### local::lib local::lib. have you heard of it? Are you using it? Why not? Why are you still running cpan as root leaving your system perl at risk? Anything you install from vendor packages (RPM, DEB, etc) should also have it's dependencies available as vendor packages - and those (hopefully) have all been tested against other parst of the system. And besides - packages can be uninstalled. CPAN modules... it's not exactly straight-forward, nor is it fool-proof. USING Local::lib IS NOT THAT FUCKING HARD. JUST FUCKING DO IT. Look... it's pretty much automagical and there's several ways to make it work. # use it just for this one app... perl -Mlocal::lib=/path/to/libdir your_script.pl # set environment vars so it's used automatically... eval $(perl -Mlocal::lib=/path/to/libdir) # put it in your script... use local::lib qw( /path/to/libdir ); And you don't even need that path if you use the defaults. If you want cpan to install to your local lib, just use one of the first two options above. IT JUST PLAIN WORKS. I'm sure there are cpan dists that don't install under local::lib that install just fine to the system perl. Consider those dists broken. File a bug report, submit a patch, or make your own modifications... and inject your modified version into your minicpan. ### minicpan now, minicpan, CPAN::Mini, whatever you like to call it, is just plain awesome-sauce. This is really the key-stone of getting out of the death spiral and taking real control of your world. after you've installed the CPAN::Mini distribution from the CPAN, simply create an empty directory and run the minicpan command with the necessary options. Then get yourself a cup of coffee or five while the bare-minimum to create a complete working cpan mirror is constructed on your hard drive. ### mcpani similarly, mcpani, CPAN::Mini::Inject is also awesome. For whatever reason, I found the man page to be the clearest way to read the documentation and learn how to use it, but basically, you can use it to inject cpan dists you've modified, or your own cpan-like dists that you're never going to make public, or just abpout anything that can be tar-gzipped and looks like a proper cpan distribution. Use it to customize exactly what's in your minicpan. ### dpan I only looked at it briefly at first and didn't realize what dpan truly is until a second look when I started writing this talk: dpan could be the means to tie all these other pieces together way more effectively, seamlessly, and comprehansively than my slap-dash chewing-gum and duct-tape "solution". ### perlbrew I haven't used it becaus it was invented after I wrote my own scripts to automate fetching, building, and installing your own custom perl but really, it's just great. You need a perl 5.12 to test on? do this: # build and install in your $HOME perlbrew install perl-5.12.0 # make that perl the first in your $PATH perlbrew switch perl-5.12.0 # set $PATH back to normal perlbrew disable Like I said, I haven't needed to use it, but it's fairly new, and it looks like it could be very, very useful, and you can always request features from the author, or add your own and send patches :) ### git Git is the glue that binds this all together. So far, all I've done is describe tools that add layers to things but they still probably don't have enough advantages to get you using them - until you add revision control to the mix. Now, you can get lots and lots of info on how to use git at other talks so I'm not going to cover it. but think of this - your code depends on third party code. You have access to it in source form, in a format that automatically works out dependencies and sub dependencies - and builds and installs everythig in the correct order. ### minicpan on git If everything you need is in a minicpan, you can put it in a local git repository - and then you have total control. If something goes wrong, you can use all the git tools at your disposal bisect, log, rollback, even blame. ### local::lib on git furthermore, if you're installing everything to a local::lib, you can have that same level of control, if you need it! If you install something that goes horribly wrong, don't cross your fingers and hope you can use whatever means cpan gives you to uninstall - just git revert before your last change! Better yet, if you're experimenting using different sets of modules, or even building against different versions of perl, store each version's local::lib in a git branch! ### perlbrew + local::lib + minicpan + git, oh my! Something I haven't yet done, but I'd likje to experiment with is to tie together perlbrew with local::lib + minicpan + git, where when you use perlbrew to switch to using a different build of perl it also switches to the apropriate branches in your local::lib and your minicpan... I haven't done it yet, but patches are welcome! #################################################################### ### Tools yet to come ### BackPAN::Version::Discover Scans your perl inc directories and tries to figure out not just what cpan distributions you have installed, but exactly what versions, even if it's no longer on the cpan... then gives you the paths to the dist files on a backpan mirror... which you can then download and use to build your minicpan/dpan! ### I went a little crazy Admittedly, I went a little crazy when I wrote it, but I think some of us have found that a little insanity can go a long way ### [ Damian, head ] Originally I was going to use this pic of him ### [ Damian, dunk-tank ] But I wasn't sure if it would be considered good taste :) ### [ Damian, looking angry ] ### DOUBLING BACK So, what about modules that won't install? If they have broken build scripts or broken tests, chances are you should find something else. seriously. either that, or file bugs and submit patches but first... Fix it yourself. Increment the version, run make dist, then inject the new tarball into your minicpan. PROBLEM SOLVED. Go back, make patches, and send to the author. What about being stuck trying to update stuff because your perl is too old? If you've built a minicpan with everything you need that works on your current version, try using perlbrew to experiment with newer perls. Either that, or experiment with injecting older versions of the things you need until everything does work. I know that it's a bit more complicated than that, but hopefully somebody will figure out a simple, streamlined way to do this and share it with the community as well. |