Channels ▼
RSS

Web Development

Private CPAN Distributions


brian has been a Perl user since 1994. He is founder of the first Perl Users Group, NY.pm, and Perl Mongers, the Perl advocacy organization. He has been teaching Perl through Stonehenge Consulting for the past five years, and has been a featured speaker at The Perl Conference, Perl University, YAPC, COMDEX, and Builder.com. Contact brian at comdog@panix.com.


In the January 2004 issue, I wrote about "CPAN in the Desert." I was using a "minicpan" (a minimal, local copy of the Comprehensive Perl Archive Network) to install modules. Since then, the "private" minicpan idea has gone even further and I can now add my own modules to my local copy of CPAN, so I can install them with CPAN.pm or CPANPLUS. I can even burn my modified minicpan to a CD and give it to other people.

When I wrote the first article, I had no connection to any sort of network. I certainly didn't have space on my hard drive for all 3 GB of the full CPAN archive, so I was a bit stuck if I wanted modules I hadn't already installed. Randal Schwartz's minicpan program came to the rescue. His program downloads only the latest versions of the modules on CPAN, and it also ignores large files such as Perl distributions. Instead of 3 GB, I end up with about 400 MB: Certainly small enough to fit on a CD.

Since then, Ricardo Signes turned Randal's original minicpan script into a module, CPAN::Mini. That really doesn't mean much to most people because Ricardo's distribution comes with an updated minicpan script that does the same thing as the original, although it is more configurable. You can install CPAN::Mini and simply use the minicpan script without ever looking at the code.

Later, Shawn Sorichetti came along with another module, CPAN::Mini::Inject, that can take one of my private modules and shoehorn it into my local CPAN mirror (and it works for both the full and mini versions because they have the same structure). I find this especially handy for clients who hire Stonehenge to write private modules. Like any good Perl programmer, we reuse code from CPAN, so our work has dependencies. Our clients need to be able to install these dependencies even if they are behind a firewall or restricted from installing things from an external CPAN mirror. By giving them a CD with everything on it, and setting things up so they can install everything with CPAN.pm, we can avoid various acrobatics to get around local policy or restrictions. Indeed, for the most restrictive cases, we can even make a micro-cpan by removing everything except the stuff that we need.

To inject a module into my local repository, I need a module to start with. I'll use a fictional module, Inject::Test, which is magically ready to distribute as Inject-Test-1.01.tgz. I don't need to write a script to use CPAN::Mini::Inject because it comes with one already, named "mcpani."

Before I can use mcpani, which I can also use as a replacement for minicpan, I need to configure it. As of version 0.36, mcpani looks in one of four places for a configuration file, in this order:

  • A file pointed to by the environment variable MCPANI_CONFIG
  • $HOME/.mcpani/config
  • /usr/local/etc/mcpani
  • /etc/mcpani

On my PowerBook, where I am the only user, I choose $HOME/.mcpani/config. My configuration file is simple; see Example 1.

The "local" directive tells mcpani where the minicpan will be. This is the smaller version of the canonical CPAN. The "remote" directive tells mcpani where to fetch files for the minicpan. I want to use passive FTP because outside servers cannot create a connection to my laptop, and I want the directory permissions to be 0755. The "repository" directive tells mcpani where to put the modules I want to inject into the minicpan. The repository keeps my private stuff separate so the minicpan script doesn't lose that data when it mirrors the real CPAN and overwrites the files CPAN::Mini::Inject needs to modify.

Before I can start anything, I need a minicpan, which I'm going to put in /MCPANI. Before I can inject private modules into the local mirror, I need a local mirror. I tell mcpani to update the local mirror (which also creates it the first time), and I add the -v switch so it operates in verbose mode. This is going to take a while, so I want to see what mcpani is doing.

% mcpani --mirror -v

When this completes, I'm at the same point I was at with Randal's original minicpan script. I have a minimal version of CPAN that includes only the latest versions of the modules.

Now I want to add private modules to my mirror so I can install them as if they were on CPAN. I could do that on my own by editing a lot of files and copying archives to the right places. Although mcpani is going to do this for me, I like to know how things work.

CPAN automatically maintains several files so that tools like CPAN.pm can find the distributions. Starting from the minicpan root directory, the file ~/modules/02packages.details.txt contains entries that give the module name, the latest version, and the file location in ~/authors/id. This is how CPAN.pm finds the file it needs to install, and this is where I need to add information about modules that I want to inject into my minicpan. Example 2(a) shows the line from 02packages.details.txt for my Business::ISBN module.

Additionally, in my author directory, ~/authors/id/B/BD/BDFOY, CPAN maintains a CHECKSUMS file that has signature information about the distributions, including sizes and MD5 checksums. CPAN.pm checks these to ensure it has the right file. Before I can inject modules, I need to update this file so it contains the information for my private module. Here's the entry for my Business::ISBN module:

'Business-ISBN-1.79.tar.gz' => {
  'mtime' => '2004-12-14',
  'md5-ungz' => '82a4626a451ab68ab96c143d414ced96',
  'md5' => '318f1851ccb236136c1d1f679539e08d',
  'size' => 350871
},

Now it's time to actually inject a distribution. I have my distribution file, Inject-Test-1.01.tar.gz. First, I have to add it to my repository. That's the private staging area.

% mcpani --add --module Inject::Test --authorid BDFOY --modversion 1.01 --file Inject-Test-1.01.tar.gz

When I look in /myMCPANI, I see a new directory, authors, and a file called "modulelist." The authors directory has the same structure as the authors directory in CPAN: There is a subdirectory named "id" that contains directories for each letter of the author ID, which in turn has directories for the first two letters, and then the full ID (CPAN used to have a flatter directory structure, but now that it has about 4000 author IDs, it needs to separate them into directories). This is still just my repository though.

I haven't injected Inject::Test into my minicpan yet. I still need to move the file into my minicpan author directory and modify 02packages.details.txt and CHECKSUMS. The --inject switch to mcpani takes modules from my repository and puts them into my local minicpan.

% mcpani --inject -v
Injecting modules from /myMCPANI
/MCPANI/authors/id/B/BD/BDFOY/
Inject-Test-1.01.tar.gz ... injected

When I look in my author directory, I find that Inject-Test-1.01.tar.gz is there. Inside my CHECKSUMS file, I now have an entry for Inject::Test:

'Inject-Test-1.01.tar.gz' => {
  'mtime' => '2005-01-28',
  'md5-ungz' => '61aca20c21b0227233f6b2754a3efbe4...,
  'md5' => '0cbb2e70f7c71afa675d0c7c1b52faf0...,
  'size' => '25486'
},

In the 02packages.details.txt, I find a line for my private module, which allows CPAN.pm and other tools to install this module as if it really was on CPAN; see Example 2(b).

Once I go through all of my private modules and inject them into my local CPAN mirror, I can burn that mirror to CD (or something else) and give that to clients. On the CD, they should have everything they need to install the module and its dependencies. Since a minicpan is only about 400 MB, I have plenty of room on a standard CD to include external libraries such as expat, gd, or various other things the client may need. I don't have to worry about their network situation, firewalls, company policy, or the myriad other things that keep people from installing the things they need.

When I update, I just update the mirror, which wipes out my changes to 02packages.details.txt and CHECKSUMS. Since my private modules are in a separate place, I simply reinject everything. To automate the process, I can create my own mcpani script using CPAN::Mini::Inject.

Life with CPAN just gets easier and easier.

References

CPAN: http://www.cpan.org

minicpan: http://www.stonehenge.com/merlyn/LinuxMag/col42 .html

CPAN::Mini: http://search.cpan.org/dist/CPAN-Mini

CPAN::Mini::Inject: http://search.cpan.org/dist/CPAN-Mini- Inject

TPJ


Related Reading


More Insights






Currently we allow the following HTML tags in comments:

Single tags

These tags can be used alone and don't need an ending tag.

<br> Defines a single line break

<hr> Defines a horizontal line

Matching tags

These require an ending tag - e.g. <i>italic text</i>

<a> Defines an anchor

<b> Defines bold text

<big> Defines big text

<blockquote> Defines a long quotation

<caption> Defines a table caption

<cite> Defines a citation

<code> Defines computer code text

<em> Defines emphasized text

<fieldset> Defines a border around elements in a form

<h1> This is heading 1

<h2> This is heading 2

<h3> This is heading 3

<h4> This is heading 4

<h5> This is heading 5

<h6> This is heading 6

<i> Defines italic text

<p> Defines a paragraph

<pre> Defines preformatted text

<q> Defines a short quotation

<samp> Defines sample computer code text

<small> Defines small text

<span> Defines a section in a document

<s> Defines strikethrough text

<strike> Defines strikethrough text

<strong> Defines strong text

<sub> Defines subscripted text

<sup> Defines superscripted text

<u> Defines underlined text

Dr. Dobb's encourages readers to engage in spirited, healthy debate, including taking us to task. However, Dr. Dobb's moderates all comments posted to our site, and reserves the right to modify or remove any content that it determines to be derogatory, offensive, inflammatory, vulgar, irrelevant/off-topic, racist or obvious marketing or spam. Dr. Dobb's further reserves the right to disable the profile of any commenter participating in said activities.

 
Disqus Tips To upload an avatar photo, first complete your Disqus profile. | View the list of supported HTML tags you can use to style comments. | Please read our commenting policy.
 
Dr. Dobb's TV