Conceptual integrity in turn dictates that
the design must proceed from one mind,
or from a very small number
of agreeing resonant minds.
–Frederick Brooks Jr.
The Mythical Man Month
Perl 6 is the next major version of Perl. It´s a complete rewrite of the inter-preter, and a significant update of the language itself. The goal of Perl 6 is toadd support for much-needed new features, and still be cleaner, faster, andeasier to use.
The Perl 6 project is vast and complex, but it isn´t complicated. The projectruns on a simple structure with very little management overhead. That´sreally the only way it could run. The project doesn´t have huge cash or timeresources. Its only resource is the people who believe in the project enoughto spend their off–hours—their “relaxation” time—working to see it completed. This chapter is as much about people as it is about Perl.
The Birth of Perl 6
Back on July 18, 2000, the second day of the fourth Perl Conference (TPC4), a small band of Perl geeks gathered to prepare for a meeting of the Perl 5Porters later that day. The topic at hand was the current state of the Perlcommunity. Four months had passed since the 5.6.0 release of Perl, andalthough it introduced some important features, none were revolutionary.
There had been very little forward movement in the previous year. It wasgenerally acknowledged that the Perl 5 codebase had grown difficult tomaintain. At the same time, infighting on the perl5–porters list had grown sointense that some of the best developers decided to leave. It was time for achange, but no one was quite sure what to do. They started conservativelywith plans to change the organization of Perl development.
An hour into the discussion, around the time most people nod off in anymeeting, Jon Orwant (the reserved, universally respected editor of the PerlJournal) stepped quietly into the room and snapped everyone to attentionwith an entirely uncharacteristic and well-planned gesture. Smash! A coffeemug hit the wall. “We are *@$!-ed (Crash!) unless we can come up withsomething that will excite the community (Pow!), because everyone´s getting bored and going off and doing other things!(Bam!)” (At least,that´sbasically how Larry tells it. As is usually the case with events like this, noone remembers exactly what Jon said.)
Awakened by this display,the group started to search for a real solution.The language needed room to grow. It needed the freedom to evaluate newfeatures without the obscuring weight of legacy code. The communityneeded something to believe in, something to get excited about.
Within a few hours the group settled on Perl 6, a complete rewrite of Perl.The plan wasn´t just a language change, just an implementation change,orjust a social change. It was a paradigm shift. Perl 6 would be the community´s rewrite of Perl, and the community´s rewrite of itself.
Would Perl 6, particularly Perl 6 as a complete rewrite, have happened without this meeting? Almost certainly. The signs appeared on the lists, in conferences,and in journals months in advance. If it hadn´t started that day, itwould have happened a week later, or perhaps a few months later, but itwould have happened. It was a step the community needed to take.
In the Beginning…
Let´s pause and consider Perl development up to that fateful meeting. Perl 6is just another link in the chain. The motivations behind it and the direc-tions it will take are partially guided by history.
Perl was first developed in 1987 by Larry Wall while he was working as aprogrammer for Unisys. After creating a configuration and monitoring sys-tem for a network that spanned the two American coasts, he was faced withthe task of assembling usable reports from log files scattered across the network.The available tools simply werent up to the job. A linguist at heart,Larry set out to create his own programming language, which he called perl.He released the first version of Perl on December 18, 1987. He made it freelyavailable on Usenet (this was before the Internet took over the world,remember), and quickly a community of Perl programmers grew.
The early adopters of Perl were system administrators who had hit the wallwith shell scripting, awk,and sed. However, in the mid-1990s Perl´s audience exploded with the advent of the Web, as Perl was tailor-made for CGIscripting and other web–related programming.
Meantime, the Perl language itself kept growing, as Larry and others keptadding new features. Probably the most revolutionary change in Perl (untilPerl 6, of course) was the addition of packages, modules, and object-oriented programming with Perl 5. While this made the transition periodfrom Perl 4 to Perl 5 unusually long, it breathed new life into the languageby providing a modern, modular interface. Before Perl 5, Perl was considered simply a scripting language; after Perl 5, it was considered a full-fledgedprogramming language.
Larry, meanwhile, started taking a back seat to Perl development andallowed others to take responsibility for adding new features and fixing bugsin Perl. The Perl 5 Porters (p5p) mailing list became the central clearing-house for bug reports or proposed changes to the Perl language, with the“pumpkin holder” (also known as the “pumpking”) being the programmer responsible for implementing the patches and distributing them to the restof the list for review. Larry continued to follow Perl development, but like aparent determined not to smother his children, he stayed out of the day–to–day development, limiting his involvement to situations in which he wastruly needed.
Although you might think that the birth of the Perl 6 project would be thefirst nail in the coffin for Perl 5, that´s far from the case. If anything, Perl 5has had a huge resurgence of development, with Perl 5.7.0 released only twoweeks after the initial decision to go ahead with Perl 6. Perl 5.8, spear-headed by Jarkko Hietaniemi and released in July 2002, includes usable Uni-code support, a working threads interface, safe signals, and a significantimprovement of the internals with code cleanup, bug fixes, better documentation,and more than quadrupled test coverage. Hugo van der Sanden is thepumpking for 5.9–5.10. Plans for those releases include enhancements tothe regular expression engine,further internals cleanup and a “use perl6ish”pragma that will integrate many of the features of Perl 6. Perl 5 is active andthriving,and will continue to be so even after the release of Perl 6.0.
The Continuing Mission
Much has changed since the early days of the project. New people join thegroup and others leave in a regular “changing of the guard” pattern. Planschange as the work progresses, and the demands of the work and the needsof the community become clearer. Today the Perl 6 project has three majorparts: language design, internals, and documentation. Each branch is relatively autonomous, though there is a healthy amount of coordinationbetween them.
Language Design
As with all things Perl, the central command of the language design processis Larry Wall, the creator of the Perl language. Larry is supported by the restof the design team: Damian Conway, Allison Randal, Dan Sugalski, Hugovan der Sanden, and chromatic. We speak in weekly teleconferences andalso meet face–to–face a few times a year to hash out ideas for the designdocuments, or to work through roadblocks standing in the way of design orimplementation. The group is diverse, including programmers–for–hire, Perltrainers, and linguists with a broad spectrum of interests and experiences.This diversity has proved quite valuable in the design process, as each mem-ber is able to see problems in the design or potential solutions that the othermembers missed.
Requests for comments (RFCs)
The first step in designing the new language was the RFC (Request ForComments) process. This spurred an initial burst of community involvement.Anyone was free to submit an RFC on any subject, whether it was assmall as adding an operator, or as big as reworking OO syntax. Most of theproposals were really quite conservative. The RFCs followed a standard format so they would be easier to read and easier to compare.
Each RFC was subject to peer review, carried out in an intense few weeksaround October 2000. One thing the RFC process demonstrated was thatthe Perl community still wasn´t quite ready to move beyond the infightingthat had characterized Perl 5 Porters earlier that year.*
*Mark-Jason Dominus wrote an excellent critique of the RFC process (http://www.perl.com/pub/a/2000/11/perl6rfc.html ). It may seem harsh to people accustomed to the more open and tolerantcommunity of today, but it´s an accurate representation of the time when it was written.
Even though few RFCs have been accepted without modification, the pro-cess identified a large number of irritants in the language. These have servedas signposts for later design efforts.
Apocalypses and Exegeses
The Apocalypses* and Exegeses** are an important part of the design process.Larry started the Apocalypse series as a systematic way of answeringthe RFCs. Each Apocalypse corresponds to a chapter in his book Programming Perl,and addresses the features in the chapter that are likely to change.
*An “apocalypse” in the sense of “revelation,” not “end of the world.”
**An “exegesis” is an explanation or interpretation of a text…
However, the Apocalypses have become much more than a simple responseto RFCs. Larry has a startling knack for looking at 12 solutions to a problem,pulling out the good bits from each one, and combining them into asolution that is 10 times better than any of the proposals alone. The Apocalypses are an excellent example of this “Larry Effect.” He addresses each relevant RFC,and gives reasons why he accepted or rejected various pieces ofit. But each Apocalypse also goes beyond a simple “yes” and “no” responseto attack the roots of the problems identified in the RFCs.
Damian Conway´s Exegeses are extensions of each Apocalypse. Each Exegesis is built around a practical code example that applies and explains thenew ideas.
The p6l Mailing List
The next body of design work is the Perl 6 Language mailing list (perl6-language@perl.org), often fondly referred to as “p6l.” Luke Palmer has beendeputized as unofficial referee of the list. He answers questions that don´trequire the direct involvement of the design team or that have beenanswered before. He also keeps an eye out for good suggestions to makesure the design team doesn´t miss them in the sea of messages. The list hasapproximately 40 regular contributors in any given month, as well as a largenumber of occasional posters and lurkers. Some people have participatedsince the very beginning; others appear for a few months and move on.
Even though the individuals change, the general tone of p6l is the same. It´san open forum for any ideas on the user-visible parts of Perl 6.In the typicalpattern, one person posts an idea and 5 to 10 people respond with criti-cisms or suggestions. The list periodically travels down a speculative threadlike a runaway train, but these eventually run out of steam. Then Larry picksout the golden bits and gently tells the rest that no, he never intended Perl 6to have hyper–vulcan mechanoid scooby–dooby–doos. Even when Larrydoesn´t post, he follows the list and the traffic serves as a valuable catalystfor his thoughts.
Internals
Parrot is a grandiose idea that turned out to be more realistic than anyoneoriginally could have believed: why not have a single interpreter for severallanguages? Unlike the parent Perl 6 project,which was launched in a singleday, the plan for Parrot formed in bits and pieces over the space of a year.
On April 1,2001, Simon Cozens published an article titled “ProgrammingParrot” as an April Fools´ joke (http://www.perl.com/pub/a/2001/04/01/parrot.htm ). It was a contrived interview with Larry Wall and Guido vanRossum detailing their plans to merge Python and Perl into a new languagecalled Parrot. A few months later, when Perl 6 internals began to take anindependent path within the larger project,they dubbed the subproject“Parrot” in a fitting turn of life imitating art.
Early Steps Toward Perl 6 Internals
The earliest progress toward implementing Perl 6 started before the currentincarnation of Perl 6 was even conceived. The Topaz project, started in 1998,was spearheaded by Chip Salzenberg. It was a reimplementation of Perl 5written in C++. The project was abandoned, but many of the goals andintended features for Topaz were adopted for Perl 6 internals, and the difficulties Topaz encountered were also valuable guides.
Sapphire was another early prototype that influenced the shape of Perl 6internals. It was a one-week project in September 2000. The brainchild ofSimon Cozens, Sapphire was another rewrite of Perl 5 internals. It was neverintended for release, only as an experiment to see how far the idea could goin a week, and what lessons could be learned.
The plan for Parrot was to build a language–neutral runtime environment. Itwould support all the features of dynamic languages such as Python, Ruby,Scheme, Befunge,and others. It would have threading and Unicode support(two of the most problematic features to add into Perl 5 code) built in fromthe start. It would support exceptions and compilation to bytecode, andhave clean extension and embedding mechanisms.
The language-neutral interpreter was originally just a side effect of gooddesign. Keeping the implementation independent of the syntax would makethe code cleaner and easier to maintain. One practical advantage of thisdesign was that Parrot development could begin even though the Perl 6 language specification was still in flux.
The bigger win in the long term, though, was that since Parrot would support the features of the major dynamic languages and wasn´t biased to a particular syntax, it could run all these languages with little additional effort.It´s generally acknowledged that different languages are suited to differenttasks. Picking which language will be used in a large software project is acommon planning problem. There´s never a perfect fit. It usually boils downto picking the language with the most advantages and the least noticeabledisadvantages. The ability to combine multiple languages within a projectcould be a huge benefit. Use well–tested libraries from one language for onetask. Take advantage of a clean way of expressing a particular problemdomain in a second, without being forced to use it in areas where it´s weak.
The modular design also benefits future language designers. Instead of targeting lex/yacc and reimplementing low-level features such as garbage collection and dynamic types, designers can write a parser that targets theParrot virtual machine.
The internals development for Perl 6 falls to the Parrot project. Dan Sugalski leads the project as internals designer, and Steve Fink is the currentpumpking. The Parrot project is largely autonomous. Dan coordinates withthe rest of the design team to ensure that Parrot will be able to support thesemantics Perl 6 will require, but the language designers have very littleinput into the details of implementation. Parrot isn´t developed solely forPerl, but Perl 6 is entirely dependent on Parrot–it is the only interpreter forPerl 6.
The core communication line for the Parrot project is the mailing list, perl6-internals@perl.org, otherwise known as “p6i.” It´s a much more business-like list than p6l. Workflow in Parrot takes the form of submitted patches.Anyone is free to submit a patch, and contributors who consistently submitvaluable patches over a long period of time are granted check-in access tothe CVS repository.
Documentation
Though adequate documentation has been a goal from the very beginning,the Perl 6 documentation project is a relatively recent addition. It operatesunder the guidance of Michael Lazzaro. The stated goal of the documentation project is to systematically walk through each Apocalypse and producefully specified documentation from it. The results of the project are eventually intended to be the documentation released with Perl 6.0.
The task of the documenters is a difficult one. The specification for Perl 6 isstill in development and constantly shifting, so they´re shooting at a moving target.The process is immensely valuable though,as it helps to identifyinconsistencies or problems in the design that the broad brushstrokes of theApocalypses miss. Sometimes it is the documentation process that causesthe shift in language specification, as identified problems lead to solutionsand the solutions, in turn, trigger changes throughout the system.
Supporting Structure
Last, but not least, is the glue that holds the project together. The highestpraise belongs to Ask Björn Hansen and Robert Spier, who manage theemail, revision control, and bug–tracking systems, as well as the web pagesfor Perl 6 and Parrot. Without these systems, the project would grind to ascreeching halt.
Nathan Torkington and Allison Randal share the load of project management.Nat tends to handle outside interfacing while Allison tends to handlethe nuts and bolts of the project, but neither role is set in stone. As is typicalof open source development projects, managing the Perl 6 project is quitedifferent from managing a commercial project of the same size and complexity.There are no schedules, no deadlines, no hiring and firing, and no salaries,bonuses, or stock options. There are no employees or bosses; there isvery little hierarchy whatsoever. Management in this context isnt about giving orders, it´s about making sure everyone has what they need to keep moving forward.
In the end, it is the developers themselves who hold the project together.Each individual bears their own share of the responsibility for finding a taskthat suits their skills, coordinating with others to keep duplicated effort minimal,and making sure the job gets done.
About the Authors
Allison Randal is the assistant project manager of the Perl 6 core development team. She has been working closely with Damian Conway and Larry Wall on Perl 6 and has co written the “synopses” of Perl 6. She is dedicated to the success of the project and is one of the very first to learn about anything new that’s proposed for Perl 6.
Dan Sugalski is the chief architect for Parrot, the interpreter engine for Perl 6. He’s been a Perl 5 core developer for years, writing more than a dozen modules in the process. He’s been a contributor to The Perl Journal and The Perl Review, as well as the O´Reilly Network.
Leopold Tötsch hails from Austria where he first started working with computers in 1976. He is an independent software developer who has been exploring and developing open source software since 1991. He´s a frequent contributor to isdnlog (an open source project for monitoring ISDN lines and optimizing telephone costs) and spends the majority of his free time working on Parrot, the language-independent interpreter developed as part of the Perl 6 design strategy.
Source of this material
 |
This is Chapter 1: Project Overview from the book Perl 6 Essentials (ISBN:0-596-00499-0) written by Allison Randal, Dan Sugalski, and Leopold Tötsch, published by O’Reilly & Associates. |
To access the full Table of Contents for the book
