Beyond the descriptive comparison of the previous section, our comparisons and interviews uncovered some unexpected patterns. The first was that portals mainly target agile teams: five of those we studied said so explicitly, and none explicitly encouraged more traditional development processes. We do not believe this was due to selection bias, though the co-emergence of SourceForge (which inspired several tools) and agile methods may be a factor.
The emphasis on agile methods had a strong influence on features offered, as many of the portals were built expressly to allow agile teams to scale up and (crucially) spread out geographically. We believe this explains why they emphasize asynchronous communication (e.g., bulletin boards) over synchronous (e.g., chat).
Three of these companies also offer consulting services to help customers who aren't agile make the transition. As the Rally executive said:
You go back three years, we were doing a lot of explaining what agile is to our customers, and helping them understand why this is a good idea. There's a lot less need -- now there's a lot more of the "how".
Those same companies acknowledge that they have included some features in their tools that aren't strictly "agile" in order to support big or cross-disciplinary teams. All said that simplicity was crucial, but felt that this pragmatism reflected the growing maturity of the agile community. For example, one interviewee said people had realized that methodologies like Scrum weren't really about developing software, they were about identifying and fixing problems in software development processes. It was therefore natural for different groups to evolve different "best" practices, and for tools to follow suit.
Another noteworthy point was that almost all of the groups we interviewed acknowledged that they didn't use any well-defined process themselves (not even the Agile methods they preach). When asked, "Do you use XP?" or "Do you use Scrum?", they invariably replied that their developers used a mix of best practices that didn't strictly adhere to any published rulebook. None of the interviewees were defensive about this; all clearly believed that they had above-average developers who could be trusted to use pair programming, test-driven development, and whatever else was appropriate in context. This emphatically does not mean that their processes were chaotic: in all cases there was close and frequent coupling between development on one hand and requirements gathering and feature prioritization on the other. However, the day-to-day mechanics of actually producing high-quality code was trusted to developers and their consciences. It remains to be seen whether the users of the tools do follow specific Agile methods or, as the tool developers', they just use their own mix of Agile practices.
All of the people we interviewed also acknowledged that their own needs had been and were a major force in the tool's evolution. This is unsurprising for free open source tools like Trac, whose developers were also all users, but the vendors of commercial systems (both closed and open source) felt that their developers' experiences were "representative" enough to be mined for features. Mingle, Rally, and VersionOne also all felt that they were ahead of the curve in adopting the agile practices that their tools were intended to support, which also legitimized the recycling of their own experiences instead of following a more formal requirements elicitation process.
One clear trend in the portals we studied was providing a hosted service. SourceForge pioneered this, and only Mingle (among the recent portals), does not primarily use "software as a service" (SaaS) model ("primarily", because several commercial vendors offer a customer-hosted option, though uptake is slight). DotProject and Trac did neither primarily use this option but we believe their choice of a more traditional customer-hosted model can be attributed to them being (comparatively) older systems, and because volunteer-authored systems lack the resources to provide scalable hosting. Mingle's authors, on the other hand, believe that enterprise clients would not want to put their data on someone else's servers, though competitors such as VersionOne and Rally have demonstrated that many will.
All of the vendors who focus on SaaS claimed that not having to install and configure the portal lowered their customers' costs, and allowed them to roll out new features in small, incremental steps. Some also pointed out that it made fixing bugs (particularly security bugs) simpler and more reliable, since in-house experts could do it for almost all customers in a single step.
A corollary to being a hosted service is the use of a subscription model. Most portals charge by the month based on the size of projects, as measured by number of users and/or file upload and storage requirements. (Many of them offer free accounts for open source or non-commercial projects.) In most cases, month-by-month payments actually cost clients more than an annual subscription would, but as two interviewees independently pointed out, it's easier to get approval for a few dollars a month, with the illusion of being able to cancel, than it is to get the accounting department to sign off on a one-time expense of several hundred dollars.
We say "illusion" because none of the portals we examined makes it easy for users to export their projects for backup or use elsewhere. The contents and history of version control repositories can be relocated using third-party tools (e.g., svnadmin dump and svnadmin load for Subversion), but the tickets, wiki pages, and other content stored in portals can at best be dumped as a big blob of XML for parsing and interpretation. The high cost of switching means that customers are effectively locked in once they select a portal.
What the portals we studied don't include is equally important. None of those we studied supported modeling or user experience design, and only one (Rally) directly supported test management. Several advertised requirements management and traceability, but in practice this turned out to be nothing more than specially-labelled tickets manually linked to code or simple product backlogs. Finally, integration between portals and IDEs was weak (with IBM Jazz being a notable exception). This cannot be blamed on technology -- the Mylyn plugin for Eclipse (which integrates information from Bugzilla, Jira, Trac, and other systems) has been available for several years, and equivalent connectors for other tools would be straightforward to build. We suspect this is simply a matter of an idea not having reached its "tipping point", and predict that IDE-to-portal linkages will be commonplace within three or four years.
In the short term we expect project portals to focus on improving the functionality they already offer. As this paper was being written, for example, SourceForge announced that it will host virtualized versions of third-party open source applications so that teams can use them in a secure fashion without having to set them up. Other issues mentioned by several interviewees included scaling the portals to handle larger teams and heavier-weight development processes and better support for the needs of "non-geeks" (e.g., marketing and sales staff). We also expect to see much better support for peripheral awareness, such as the task context model provided by Mylyn [Kersten06].
In the longer term, interviewees speculated that project portals would merge or integrate with social networking tools such as LinkedIn and personal life management tools such as Google Calendar. People already put much of the information project managers require, such as skills and availability, into these systems, and are unlikely to duplicate that information manually. Thanks to the open APIs offered by most of them, web-based portals are already becoming software buses for aggregating all kinds of project-relevant data; we predict that in the end, the system that does the best job of balancing ease of configuration with privacy protection will win.
Even before this happens, portals are adopting "Web 2.0" ideas as quickly as they can. For example, most tools support alerting of changes by means of RSS feeds, many offer some kind of wiki related to the project and some portals integrate a (limited) tagging mechanism and allow search across projects based on those tags' "folk" semantics. Others have expertise advertising and rating systems to help members of an organization locate useful people [Ehrlich08]. Use of social networking may facilitate team and improve the quality and quantity of communication channels between team members [Chang07], resulting in higher-quality software [Valetto07].
Some portals are also considering the addition of modules for risk analysis, finance, and/or HR (e.g. the interviewee from Assembla pointed to the development of their recruiting system module as a distinguishing feature). Some of these extensions will likely be driven by new laws enforcing stricter policies for software development for government agencies (e.g., the stringent auditing requirements of the Sarbanes-Oxley Act in the United States). This will also drive portal vendors toward the software bus model, since organizations may need to satisfy very different jurisdictional requirements.
Finally, while this is still a young market, tools are already expanding their initial niche market (e.g. going from a specific agile process to a generic support for all kinds of agile-like processes and even to an initial support for non-agile teams) and competition for high-value niches is fierce, and we believe that it is likely that mergers (and failures) will soon leave only a small number of dominant players.
Threats to Validity
Some threats to the validity of this study have been avoided by complementing the tools' evaluations with a set of interviews with key members in the tool conception and development that gave us a better understanding of each tool and of its development and commercialization context.
Nevertheless, there are still some threats to the validity and generalizability of our conclusions that should be considered when reading this paper. First of all, the huge number of tools prevent us from studying all of them, and thus, our results are biased by the tools we finally selected. Second, there is also a bias in the opinion of all interviewees, always favourable to this kind of tools (since all of them were directly involved in their commercialization/development). An additional possible bias is the fact that we did not formally include in the study the opinions of the tools' users (beyond our own experience and the informal feedback comments from colleagues we contacted in different kinds of companies and locations). (Both authors are active users of this kind of tools and one author (Wilson) created a mini-portal suitable for classroom use based on Tract.)
Despite these potential biases, we believe that our tool selection process and the key role of our interviewees in their companies makes our results representative enough to be useful for all software engineers thinking in adopting a web-based project management portal in their new development project.
Web-based project management portals are now the heart of many mature software development projects. In this paper we have presented the results of our study of these tools' origins, features, use, and likely evolution. The most noteworthy results are the fact that the creators of tools meant to support agile processes do not strictly adhere to those practices themselves, but instead trust their developers to use good practices when and as their own best judgment dictates, and the general lack of support for non-code-centric activities such as requirements management, modeling, user experience design, and test management.
Both observations raise questions about the focus and direction of much current software engineering research. In particular, we now wonder about the real importance of requirements elicitation and structured development process in the success of a development project (at least when all team members can be classified as expert software engineers). We also wonder if other niches of tools are developed in a similar way.
Other possible research directions derived from this work could focus on understanding how these portals are really used. For example, it would be interesting to analyze the ways in which ticketing systems are configured by their users (i.e. what information do they track, who (and when) enters that information, whether the information in the portal is enough to carry out the tasks or teams still rely on personal communications not electronically recorded in the portal, and so forth) in order to learn more about their actual development processes. This could influence how the next generation of portal tools is shaped to better satisfy the real users' needs.
We wish to thank everyone who agreed to be interviewed for this study, and apologize in advance for any over-simplifications in this paper. Jordi Cabot's work was supported by the Catalan Government (Grant 2007 BP-A 00128) and the Spanish Ministerio de Ciencia y Tecnologia (TIN2008-00444 project).
This document is made available under the Creative Commons Attribution license. You are free:
- to Share -- to copy, distribute and transmit the work
- to Remix -- to adapt the work
Under the following conditions:
- Attribution -- You must attribute the work in the manner specified by the author or licensor (but not in any way that suggests that they endorse you or your use of the work).
With the understanding that:
- Waiver -- Any of the above conditions can be waived if you get permission from the copyright holder.
- Other Rights -- In no way are any of the following rights affected by the license:
- Your fair dealing or fair use rights;
- The author's moral rights;
- Rights other persons may have either in the work itself or in how the work is used, such as publicity or privacy rights.
- Notice --- For any reuse or distribution, you must make clear to others the license terms of this work. The best way to do this is with a link to this web page.
For the full legal text of this license, please see this page.
[Ashawi03] Mustafa Alshawi and Bingunath Ingirige: "Web-enabled project management: an emerging paradigm in construction". Automation in Construction, 12(4), 2003. DOI: 10.1016/S0926-5805(03)00003-7.
[Chang07] Klarissa Chang and Kate Ehrlich: "Out of sight but not out of mind?: Informal networks, communication and media use in global software teams". CASCON 2007 http://doi.acm.org/10.1145/1321211.1321221.
[Ehrlich08] Kate Ehrlich and N. Sadat Shami: "Searching for Expertise". CHI 2008, http://doi.acm.org/10.1145/1357054.1357224.
[IEEESoftware08] Simon Helsen, Arthur Ryman, and Diomidis Spinellis: "Software Development Tools: Guest Editors' Introduction". IEEE Software, 25(5), 2008.
[Kersten06] Mik Kersten and Gail C. Murphy: "Using task context to improve programmer productivity" SIGSOFT FSE 2006, http://doi.acm.org/10.1145/1181775.1181777.
[Scrum] Ken Schwaber: Agile Project Management with Scrum. Microsoft Press, 2004, 978-0735619937.
[Terry98] Terry L. Fox and J. Wayne Spence: "Tools of the Trade: A Survey of Project Management Tools". Project Management Journal, 29(3), 1998.
[Valetto07] Giuseppe Valetto, Mary E. Helander, Kate Ehrlich, Sunita Chulani, Mark N. Wegman, and Clay Williams: "Using Software Repositories to Investigate Socio-technical Congruence in Development Projects". MSR 2007, http://doi.ieeecomputersociety.org/10.1109/MSR.2007.33.
[XP] Ken Beck and Cynthia Andres: Extreme Programming Explained: Embrace Change. Addison-Wesley Professional, 2004, 978-0321278654.