|
Creating private module distributions in a miniature CPAN repository can be a great way to make sure clients have all their module dependencies taken care of.
February 01, 2005
URL:http://www.drdobbs.com/web-development/private-cpan-distributions/184416190
brian has been a Perl user since 1994. He is founder of the first Perl Users Group, NY.pm, and Perl Mongers, the Perl advocacy organization. He has been teaching Perl through Stonehenge Consulting for the past five years, and has been a featured speaker at The Perl Conference, Perl University, YAPC, COMDEX, and Builder.com. Contact brian at [email protected].
In the January 2004 issue, I wrote about "CPAN in the Desert." I was using a "minicpan" (a minimal, local copy of the Comprehensive Perl Archive Network) to install modules. Since then, the "private" minicpan idea has gone even further and I can now add my own modules to my local copy of CPAN, so I can install them with CPAN.pm or CPANPLUS. I can even burn my modified minicpan to a CD and give it to other people.
When I wrote the first article, I had no connection to any sort of network. I certainly didn't have space on my hard drive for all 3 GB of the full CPAN archive, so I was a bit stuck if I wanted modules I hadn't already installed. Randal Schwartz's minicpan program came to the rescue. His program downloads only the latest versions of the modules on CPAN, and it also ignores large files such as Perl distributions. Instead of 3 GB, I end up with about 400 MB: Certainly small enough to fit on a CD.
Since then, Ricardo Signes turned Randal's original minicpan script into a module, CPAN::Mini. That really doesn't mean much to most people because Ricardo's distribution comes with an updated minicpan script that does the same thing as the original, although it is more configurable. You can install CPAN::Mini and simply use the minicpan script without ever looking at the code.
Later, Shawn Sorichetti came along with another module, CPAN::Mini::Inject, that can take one of my private modules and shoehorn it into my local CPAN mirror (and it works for both the full and mini versions because they have the same structure). I find this especially handy for clients who hire Stonehenge to write private modules. Like any good Perl programmer, we reuse code from CPAN, so our work has dependencies. Our clients need to be able to install these dependencies even if they are behind a firewall or restricted from installing things from an external CPAN mirror. By giving them a CD with everything on it, and setting things up so they can install everything with CPAN.pm, we can avoid various acrobatics to get around local policy or restrictions. Indeed, for the most restrictive cases, we can even make a micro-cpan by removing everything except the stuff that we need.
To inject a module into my local repository, I need a module to start with. I'll use a fictional module, Inject::Test, which is magically ready to distribute as Inject-Test-1.01.tgz. I don't need to write a script to use CPAN::Mini::Inject because it comes with one already, named "mcpani."
Before I can use mcpani, which I can also use as a replacement for minicpan, I need to configure it. As of version 0.36, mcpani looks in one of four places for a configuration file, in this order:
On my PowerBook, where I am the only user, I choose $HOME/.mcpani/config. My configuration file is simple; see Example 1.
The "local" directive tells mcpani where the minicpan will be. This is the smaller version of the canonical CPAN. The "remote" directive tells mcpani where to fetch files for the minicpan. I want to use passive FTP because outside servers cannot create a connection to my laptop, and I want the directory permissions to be 0755. The "repository" directive tells mcpani where to put the modules I want to inject into the minicpan. The repository keeps my private stuff separate so the minicpan script doesn't lose that data when it mirrors the real CPAN and overwrites the files CPAN::Mini::Inject needs to modify.
Before I can start anything, I need a minicpan, which I'm going to put in /MCPANI. Before I can inject private modules into the local mirror, I need a local mirror. I tell mcpani to update the local mirror (which also creates it the first time), and I add the -v switch so it operates in verbose mode. This is going to take a while, so I want to see what mcpani is doing.
% mcpani --mirror -v
When this completes, I'm at the same point I was at with Randal's original minicpan script. I have a minimal version of CPAN that includes only the latest versions of the modules.
Now I want to add private modules to my mirror so I can install them as if they were on CPAN. I could do that on my own by editing a lot of files and copying archives to the right places. Although mcpani is going to do this for me, I like to know how things work.
CPAN automatically maintains several files so that tools like CPAN.pm can find the distributions. Starting from the minicpan root directory, the file ~/modules/02packages.details.txt contains entries that give the module name, the latest version, and the file location in ~/authors/id. This is how CPAN.pm finds the file it needs to install, and this is where I need to add information about modules that I want to inject into my minicpan. Example 2(a) shows the line from 02packages.details.txt for my Business::ISBN module.
Additionally, in my author directory, ~/authors/id/B/BD/BDFOY, CPAN maintains a CHECKSUMS file that has signature information about the distributions, including sizes and MD5 checksums. CPAN.pm checks these to ensure it has the right file. Before I can inject modules, I need to update this file so it contains the information for my private module. Here's the entry for my Business::ISBN module:
'Business-ISBN-1.79.tar.gz' => {
'mtime' => '2004-12-14',
'md5-ungz' => '82a4626a451ab68ab96c143d414ced96',
'md5' => '318f1851ccb236136c1d1f679539e08d',
'size' => 350871
},
Now it's time to actually inject a distribution. I have my distribution file, Inject-Test-1.01.tar.gz. First, I have to add it to my repository. That's the private staging area.
% mcpani --add --module Inject::Test --authorid BDFOY --modversion 1.01 --file Inject-Test-1.01.tar.gz
When I look in /myMCPANI, I see a new directory, authors, and a file called "modulelist." The authors directory has the same structure as the authors directory in CPAN: There is a subdirectory named "id" that contains directories for each letter of the author ID, which in turn has directories for the first two letters, and then the full ID (CPAN used to have a flatter directory structure, but now that it has about 4000 author IDs, it needs to separate them into directories). This is still just my repository though.
I haven't injected Inject::Test into my minicpan yet. I still need to move the file into my minicpan author directory and modify 02packages.details.txt and CHECKSUMS. The --inject switch to mcpani takes modules from my repository and puts them into my local minicpan.
% mcpani --inject -v Injecting modules from /myMCPANI /MCPANI/authors/id/B/BD/BDFOY/ Inject-Test-1.01.tar.gz ... injected
When I look in my author directory, I find that Inject-Test-1.01.tar.gz is there. Inside my CHECKSUMS file, I now have an entry for Inject::Test:
'Inject-Test-1.01.tar.gz' => {
'mtime' => '2005-01-28',
'md5-ungz' => '61aca20c21b0227233f6b2754a3efbe4...,
'md5' => '0cbb2e70f7c71afa675d0c7c1b52faf0...,
'size' => '25486'
},
In the 02packages.details.txt, I find a line for my private module, which allows CPAN.pm and other tools to install this module as if it really was on CPAN; see Example 2(b).
Once I go through all of my private modules and inject them into my local CPAN mirror, I can burn that mirror to CD (or something else) and give that to clients. On the CD, they should have everything they need to install the module and its dependencies. Since a minicpan is only about 400 MB, I have plenty of room on a standard CD to include external libraries such as expat, gd, or various other things the client may need. I don't have to worry about their network situation, firewalls, company policy, or the myriad other things that keep people from installing the things they need.
When I update, I just update the mirror, which wipes out my changes to 02packages.details.txt and CHECKSUMS. Since my private modules are in a separate place, I simply reinject everything. To automate the process, I can create my own mcpani script using CPAN::Mini::Inject.
Life with CPAN just gets easier and easier.
CPAN: http://www.cpan.org
minicpan: http://www.stonehenge.com/merlyn/LinuxMag/col42 .html
CPAN::Mini: http://search.cpan.org/dist/CPAN-Mini
CPAN::Mini::Inject: http://search.cpan.org/dist/CPAN-Mini- Inject
TPJ