I’ve given a talk in the recent Open Forges Think Tank track of Open World Forum, which was organized by Christian Rémy from Bull, also a partner in the COCLICO project (btw, thanks Christian, this was a great track, with several interesting presentations and a great panel).
I’ve had the privilege to speak on behalf of the whole COCLICO project, in the afternoon session which was focused on forges interoperability.
This article will somehow be a transcript of what I’ve said (or intended to say), with the accompanying slides available here.
In this quite long piece, I’ll first recap some of the context elements about the COCLICO project. Then I will describe the interoperability issues that I’ve tried and focused on in my presentation, including the issues of project lock-in in the forges. I’ve tried also to describe the current ideas we’ve elaborated in the project to address these issues of interoperability (including our plans for open standards elaboration for forges interoperability). I finally conclude with a proposal to join the PlanetForge community for all interested parties.
Unfortunately, not all of these ideas are currently yet properly documented on the COCLICO website, so I hope this article will serve as a useful reference for what COCLICO is doing, still being a subjective piece of my own views, not necessarily representing those of other COCLICO participants, nor a precise description of what we’ll manage to achieve in COCLICO or PlanetForge.
The COCLICO project is addressing many different topics about the software development forges, but interoperability issues are present in many of its workpackages. So I hope our views can help shed some light on current problems, and that we will be able to propose promising developments to address these in a concrete manner.
Right before me in the track, there was a very interesting round table/panel discussion which exhibited many different aspects of forges interoperability (I’ll try to summarize the discussions in another installment). One of the major problems that was exhibited in that discussion, is that there is a lack of standards that could help address the different interoperability issues, for instance, the lack of a standard format for backup/dump/export of forge projects data (but same stands for dynamic interoperability between tools).
After a very inspiring panel discussion for most participants (I hope), I’ve tried to provide some more links illustrating some of the elements that were discussed and on which the COCLICO project has been working for a year now. Most are big concerns for me at the moment, so hopefully my slides’ content and the ideas exposed by panel participants were quite complementary.
Also, I tried and provided a few ideas about the potential of some emerging (in the forges domain) standards or techniques that may prove helpful in solving interoperability issues. Among these is the use of the OSLC standard, which was the topic of the next speaker’s presentation (nicely devised schedule of talks).
About the COCLICO project
Those impatient readers or those who already heard about the COCLICO project may skip this part.
COCLICO is a 2 years project funded under the french collaborative R&D cluster programmes (pôle de compétitivité) System@tic (Paris Region) and Minalogic (Grenoble Region), which is arriving at mid-term now (fall 2010).
Our consortium groups 9 participants mainly in Paris and Grenoble, with several large industrial companies (Bull, Orange Labs, Xerox), SMEs (CELI France, Bearstech, Gnurandal and Objet Direct) together with two academic institutions (INRIA and Institut TELECOM, more precisely our team at Telecom SudParis). Most have past expertise in the forges domain.
The COCLICO project has a strong focus on Free/Libre/Open Source software (FLOSS) as per the charter of the funding programme System@tic’s special interest group on FLOSS (GTLL). IMHO, that should sound natural whenever public spending subsidies software development, anyway.
The goals of the project are (excerpt from our site) : “The Coclico project aims to reinforce software forges communities by structuring an open source ecosystem for which a critical mass exists in France.”
That said, we’re aiming international impact, of course, at least for what relates to technology, hoping that only the market development aspects will be substantially better in France than anywhere else, if we’re smart enough .
More precisely, the COCLICO Project’s goals include (again repeating the website) :
- Re-dynamization of the Open Source development community around open source historic forge code base (FusionForge and Codendi)
- Definition of an open integration model
- Data integrity and confidentiality
- Exchange of data in real-time between various forges
- Features for industrial use and quality assurance
- traceability of information,
- support of software engineering methodologies,
- interaction with the user’s workstation.
Interoperability issues and project lock-in
Let’s get back to the current interoperability issues of software development forges.
Forges have been a blessing for many communities, both on the Net, and inside organizations. They helped solve a great deal of problems, but they show some limitations, and their use brings new risks.
Among these risks, I can see one major problem, namely the projects lock-in to their hosting forges. It is not the only problem users or forge owners are facing, and definitely not one that may deserve everyone’s attention, compared to other issues that may be experienced in everyday forges use. But I think that too much locking of projects data into the tools used to host the project can have damaging consequences. It may not be perceived as a major problem by most but has definitely been a problem for some users at particular instants in their communities history, and thus deserves that we try and address it the best way.
When users are not satisfied with the tools they use and they have little hope to be able to change tools easily, this generates enough frustration and anger to have a substantial impact on the quality of the community relations, on the motivation and productivity of the projects. This is even more important for FLOSS communities, and in general all communities of volunteers, which rely mainly on the good will of all participants, where great care must be taken to preserve the cohesion, and to avoid that tools become a problem rather than a means.
Whenever forges don’t allow projects to adapt the way they use the tools to their needs and the evolution of their community, forges can harm these projects. And moving a project from one forge to another one, for instance because new tools are available on the new one, isn’t easy nor exactly possible in most cases today.
So even if projects are happy with the current tools they’re using on one particular forge, they should have the option to change those tools at any point in time without fearing too much disturbance.
I see the possibility of changing hosting facilities for projects/communities as a fundamental freedom that should be preserved, even if there is not the everyday need to exercise such freedom. As such, interoperability of forges in such aspects can become as important as the legal protection that FLOSS licenses offer to such projects.
Not everyone has the need to fork a project, or patch a source code, but licenses guarantee that one may do it should it become necessary, for FLOSS users and developers. And the same reasoning should apply for the freedom to fork a community, for instance, so that a community has the means to leave a project hosted on one forge, and create a new project, taking with it the artifacts (data and meta-data) that constitute this online community, without having to start again from scratch. Such needs are rare, but do happen in FLOSS communities.
It is ultimately the responsibility of forge owners to preserve such freedom for the users of their tools, and for the communities hosted on their forges.
Making backups doesn’t mean that you’ll crash data and need to restore, but not making backups surely won’t help in the event that a crash occurs.
Who’s backing up community relationships, and all the history of projects that have been woven through collaboration on the many tools hosted on a software forge, then ?
Then, having the possibility to export/backup contents of all the artefacts and relations that constitute a project in a forge is quite important. Even more important is the ability to restore these into another set of tools on another forge. Technical and semantical interoperability then enters the field at this point.
Unfortunately, at the moment there hasn’t been any standard format proposed for such exports or backups to be made. That’s one of the problems we’re trying to address in the COCLICO project.
As I said before, this is not the only interoperability issue that forge users and owners face, but it is surely one worth tackling. Hopefully, by trying to solve this problem and others, common elements of a solution can appear, leading to broader interoperability solutions and general standards.
Who’s caring actually ?
Those who have been in the forges business for a long time may remember this project from 2003, the CoopX initiative, which never actually delivered, but was the first public attempt to try and work on a standard for export of forges data, to my knowledge.
Now, in 2009-2010, the subject seems to be somewhat still unsolved, but still a problem for people.
For instance, Eric S. Raymond (ESR) blogged about “Three Systemic Problems with Open-Source Hosting Sites“, where he draws a very bad situation in that :
- Hosting Sites Are Data Jails
- Hosting Sites have Poor Scriptability and finally
- Hosting Sites Have Inadequate Support for Immigration (in the provicative style of ESR).
Another prominent voice (at least in the Perl community), Jesse Vincent tries to warn us against the problems of hosted infrastructures for users who lose control over their tools. The bottomline of his speech (“Web 2.0 is Sharecropping” conference) is “If you don’t own your tools, you’re going to be in a whole mess of trouble.“. Said by the CEO of a bugtracker hosting company, that seems quite believable, I guess
Another influential voice these days must be heard also in this part of the problem analysis, I think, namely prof. Eben Moglen who warns us against the far too large power gathered by the owners of the social web sites and the owners of the cloud (see for instance his conference “Freedom in the Cloud“. After all, what’s differentiating a software development community from any other community of “social networking” websites users ?
Forges can become jails for our communities of developers and users in a maner similar to those at Facebook, twitter and other sites… but fortunately on a lower scale, since the network effects are less important here. Also, the importance of privacy issues is lower, but still, too much concentration and lock-in should be fought against, maybe with complementary techniques and tools.
But these ideas, that have been present in the mind of the proponents of software freedom since more than a decade, are not always familiar to those who work in the forges area (and I acknowledge there may be more urgent matters). Still, these lockin and hosting issues happen from time to time, either for a single project leader (like ESR with
gpsd and BerliOS) or for a larger community.
One of the latest such hazards I’ve noticed, is stated in the FAQ of the newly created Document Foundation that established itself in order to give the OpenOffice.org community a more independant structure (and a new name for the software : LibreOffice).
It goes like this (http://www.documentfoundation.org/faq/ as of sept. 30 2010):
Q: Why are you building a new web infrastructure?
A: Since Oracle’s takeover of Sun Microsystems, the Community has been under “notice to quit” from our previous Collabnet infrastructure. With today’s announcement of a Foundation, we now have an entity which can own our emerging new infrastructure.
I’m not so sure about the details (comments welcome), but I can feel the additional burden on the newly born foundation, to have to setup a new hosting infrastructure and copy part of the knowledge base that its community has built over years.
There are many hosting services for communities wishing to establish their development infrastructure somewhere on the Net. And public hosting sites are not the majority of such forges, with the even larger number of self-hosted communities and projects.
There happens to be a growing number of forge software offerings (many as FLOSS), to be installed, and offering different variants of community support and software developement tools.
Still, even though it is ever easier to choose a forge at a project’s birth, it has not become easier to move one project from one forge to another over years. Maybe the situation improves a little bit with the advent of distributed version control tools such as Git or Bzr which natively support cloning of a whole repository of source code. Still, other than for source code, it’s still a pain to move a mailing list or a forum for instance (not to speak about release systems / download areas).
It is great to see many new forges appearing every month, but it would even be greater if they did not reproduce the errors of their predecessors, and would some day support a way out for their hosted projects, preserving users freedom of change.
Just in case, to make it clear, COCLICO is not going to reinvent yet another forge. Actually, instead, we try and work at reducing forges fragmentation, by allowing some convergence of implementations between FusionForge and Codendi, on the plugin system for instance. Both Codendi and FusionForge have a common code-base ancestry in the venerable PHP code of the original SourceForge dating circa 2000. So it is quite feasible to share some PHP code and internal libraries, the most basic way to provide interoperability
Due to the French context and the natural pre-existing links among participants to the project, FusionForge has been targetted from the very beginning as a primary receptacle for many of the COCLICO efforts. FusionForge’s lead developers live and work in France, and a good share is onboard COCLICO. FusionForge has thus benefitted from the added momentum that the COCLICO launch represented for its former users. Many of those that had previously deployed GForge instances, and that had customized them to integrate with their IT, were afraid of the lack of momentum of the GForge community, and feared there would be no way to preserve their investment in that forge. The birth of the FusionForge community with the goal of re-animating the development, and the launch of COCLICO, which would be active in bringing new efforts around FusionForge, helped a lot of other parties to rally the band. Indeed, the pace of development (illustrated by commits to the code) has dramatically increased since COCLICO was started.
We also work on implementing an import/export toolbox and a standard dump format for forges, in order to try and solve the lock-in issue (based on an extended version of ForgePlucker; more about this later).
We also try to study and provide some bits of implementation in order to help users of the venerable forges to inter-operate wit more modern tools, or in more modern contexts. This includes :
- more REST APIs (around OSLC-CM for instance, for the trackers)
- more modern semantic representation formats for data hosted in the forges (as RDF or RDFa, which provides inherent extensibility)
- we investigate (based on these REST and RDF Web techniques) the possibility of more Linked Data oriented tools in the forges (see below)
- we also work on integrating new tools and methodologies (Continuous Integration, tests, Eclipse, etc.)
Another important aspect of our activity is to raise awareness of these issues in the community of forge owners/providers and users. Events such as the forges think tank at OWF 2010 or the Forges track at RMLL/LSM 2010 are examples of this effort.
As physical meetings and workshops are not enough and contacts and work needs to be continued online, we established a proposed online community “PlanetForge” to try and reach as many interested parties as possible (see bellow). Of course, the goal is to have much more participants than just the COCLICO partners in PlanetForge, so that our dissemination and standardisation efforts can meet others’ and a lively community can work together in a lasting manner.
Data portability seems a very interesting initiative whose goal is to solve such lock-in problems like the one we exhibited for the forges in this piece.
However it is not obvious to us if it may be the right place for discussing Forge data portability. Maybe joint efforts could be made between us (in COCLICO and the PlanetForge community) and participants to DataPortability.
Initial contacts seemed to produce no real echo, but that may just be caused by bad timing or misunderstanding. In any case, we’re very open to any more contacts with the initiative, should anyone help with such contacts (feel free to comment bellow this article).
ForgePlucker, a tool developed by Eric S. Raymond (ESR), is interesting in many aspects.
Its goal is to help users locked-out of their project, to regain access to data hosted on a forge. Of course that is just one of the use cases where data export from a forge needs to be performed, but certainly a valid one (see the above mentioned blog post and the forgeplucker announcement by ESR for the very detailed rationale).
There’s code available in ForgePlucker’s repository, but that would still require a lot of work (and many improvements) in order to provide a full backup of a forge project. And of course developing connectors for all the many forge variants would require much effort, not to mention the importers for all different target forges.
Our plan in COCLICO is to extend and improve the current core of ForgePlucker, to produce an Export / Import toolbox that may become generic enough to solve the lock-in issues for many users (still requiring them to contribute to it some custom connectors).
The format of the dumps produced by the current initial version of ForgePlucker is JSON, but with a non-semantic ad-hoc content. We intent to improve this to render it more semantic, by including reference to particular ontologies, with the RDF model.
Whenever possible we’d then use existing ontologies and standards instead of ForgePlucker’s own custom ontology, such as FOAF, DC, DOAP, or OSLC core (for OSLC, see below).
Linked Open Social Web
We intend to use the same approach as for the dump format, of using the RDF model to describe forges data and meta-data, reusing and extending existing ontologies, together with REST interfaces to demonstrate how it is possible to include such data in Semantic Web formats in the very interfaces of the forges.
The goal will be to make sure that development artifacts and communities that are hosted in (public) forges can become first class resources participating the Linked Open Data cloud (more details about the Linked Data approach on http://linkeddata.org/)
We intend to see this leveraged by a similar approach adopted in the OSLC compatible tools (see bellow).
In order to achieve such goals, we propose an ontology that represents data/meta-data and associated links, in a way which models the existing structure of tools and data found in forges.
We intend to propose this ontology to extend OSLC data representation standards whenever suitable, to make forge resources available to the Linked Open Data cloud.
Of course, many artifacts found in the forges resemble those found in other social sites or collaboration platforms, mainly when it relates to the people participating on the sites, as groups or communities, discussing and solving problems, sharing content together.
It is interesting to note that in the same way as development becomes more distributed (following the example of the FLOSS projects), and provides new tools supporting such distributed work (as Git emerges vs the centralized model imposed by Subversion (SVN)), social networks become more and more distributed also. Of course, solving the lock-in issues can be done through intrinsic distribution.
As SVN loses ground against Git or Bazaar (BZR), traditional RDBMS see No-SQL database systems emerging, and some predict that FaceBook may see a competitor in Diaspora, the tendency may be for more and more social sites and hosting platforms (forges included) to become more distributed, more inter-linked, more Social Semantic Web (Web 3.0), and so “no-forges” may be the future ?
We can see already some interesting attempts in such directions, with projects like SD, Fossil, or the new Savane rewrite, as well as the next generation forges proposed by QualiPSo, and on which COCLICO participants are working also nowadays.
Still, more traditional (monolithic) forges are in great need of dynamic interoperability, to allow the plugging of tools to render new services around the development activities.
Another aspect on which COCLICO is actively working is dynamic interoperability of forges with other tools.
Some emerging standards are worth citing as we try and implement support for them in existing forges like FusionForge and Codendi in COCLICO.
First, Open Services for Lifecycle Collaboration (OSLC) which is a proposed standard, elaborated as an Open standard mainly by commercial editors in the ALM sector. Despite its origin that may trigger suspicion from hard-core FLOSS addicts, it appears to be a very nice open standard, elaborated by an open community, and with, hopefully, many good technical and methodological principles. Thus, we see it as a very good candidate for establishing an interoperability standard in the forges domain.
About the technological aspects, it is notable that all specs are based on the use of Web standards, like REST, RDF, or Ajax interactions.
We have already started implementing OSLC-CM, the Change Management domains specification of OSLC for the Mantis bugtracker in the prior HELIOS project. We continue these efforts in order to implement an OSLC-CM compatible REST server for FusionForge trackers (reusing the same codebase as for Mantis, also a PHP app).
We do hope that OSLC Core, the most basic specs that underline all the other domain-specific OSLC specs can be a building block for future forges-related specifications and standards. Of course it seems reasonable to also contribute to OSLC whichever standardization elements come from our experiments in the forges area in COCLICO, as we had already started to do during HELIOS for the OSLC-CM V2 specs. At least, at Institut TELECOM, we intend to continue to contribute to OSLC specs, and implement the standard in FLOSS applications.
Some other standards or specifications seem quite interesting and we are going to try and support them in forges during COCLICO, like WebID (aka FOAF+SSL), or OAuth (which is recommended by OSLC, by the way).
We do hope that other participants to the PlanetForge community will join us in testing, validating, and promoting such standards for greater interoperability of the products they use or develop.
The PlanetForge community
Finally, as was aready mentioned several times above, we hope to help all users, editors, implementors, and owners/admins of forges to exchange and collaborate more, mainly regarding standardization, for instance in order to preserve users freedoms, as well as to better integrate different tools with each-other.
We propose to host such inter-forges and inter-communities collaboration around forges under the PlanetForge umbrella, hosted at http://planetforge.org/.
At the moment, the following on-line tools are available :
- a News agregator (planet RSS agregator) at http://planet.planetforge.org/
- a microblogging hashtag/community /
- the discussion mailing-list in english : email@example.com
- a french speaking mailing-list also
- a wiki (running MediaWiki) to share content (specs, use cases, useful pointers) at : http://wiki.planetforge.
In addition, we intend to organize, again under this PlanetForge umbrella, series of events allowing direct physical contact between all participants, in the line with previous meetups held at events like RMLL/LSM or OWF.
We do hope people interested by the subject of forges will participate to PlanetForge, exchanging and sharing ideas, experience, collaborating around tools, and of course participating to standardization efforts for more interoperability.
I hope this long article will have provided a few useful bits of information, and some hints on how the current interoperability issues of the forges may be addressed properly, and hopefully an interesting description of what COCLICO is trying to do in this respect. I sincerely hope others will join our efforts in the frame of the PlanetForge community, for the larger benefit of all actors.
Special thanks to Madhumita for english proofreading.