What's the deal with Apache?
May 14th, 2007 by HenThis has been something I’ve been pondering in the last couple of days. What’s the big deal with the arrogant, stuck up, crap quality, absorb the world Apache Software Foundation?
At first my thinking was ‘Enterprise’. I’ve blogged on this before I think; basically Apache is an experiment in lumping together large amounts of open source projects into a single entity. Much in the way that your typical large computer company is the amalgram of many smaller departments who could all be their own small companies; or that Wal-Mart is the amalgram of many small stores and a whole series of transport groups (he says with no knowledge of how Wal-Mart works). That doesn’t explain much though, just that it’s a bit of an amorphous blob, in many ways like SourceForge, Java.Net, GoogleCode and others.
There’s a difference of course. The three I mentioned above are more like shopping malls than a single company. There’s no common thread across their departments. I can walk from one project to the next and experience entirely different cultures, life, style or conversations. Within Apache there is an underlying meme that ties these projects into a single entity.
Ted Husted has attempted to capture that meme in his Apache rules. It’s not a bad stab at listing things. The first and foremost is that Apache release things under the Apache license, an evolution of the BSD/MIT style license which is an attempt at controlled public domain. I think you can understand a lot about Apache from this. The license gives huge amounts of freedom to the user, and the Apache community mindset is one of low levels of competition. It happens, we’re all competitive, but it’s much harder to be competitive with a BSD-like license, plus the community reinforces that through the shared memes. Competition is good for evolving a better product, wanting to defeat the competition is bad. We’re still individuals at the end of the day, and individuals are hugely competitive so you can only take that so far.
Why are Apache projects more successful than others? Obviously a badly worded question as who knows if they are. Let’s try again. Why do people choose an Apache project ahead of a non-Apache one? Why does an inferior project still grab users when a superior non-Apache one sits and wonders why the users aren’t migrating?
It’s all about brand - and the common meme above. Apache Foo means something. It means that Foo, whatever Foo is, shares the memes above. Without even knowing what Foo is, how well it’s coded or what it does, a user (or a user’s boss, or a CEO) can know something about Foo. That’s what Apache as a brand gives us. There are other brands out there that do the same. JBoss Foo tells a user about the Foo; Spring Foo tells a user about the Foo. To a lesser extent I think, Codehaus Foo, OpenSymphony Foo and ObjectWeb Foo tell a story [lesser because I think their common memes are are not as strong or not as long a list]. So yes, if I’m going to look at FooTechs, I’m going to pick up the one whose meme I know and want to use.
As an aside. I think it’s interesting that Apache, JBoss and Spring are all individual products whose brand has been used to create an enterprise collation of open source projects. I wonder if that played any part in their successes? Did it hamper Codehaus/OpenSymphony/ObjectWeb?
The Apache Incubator of course is a system for certifying that said memes are being followed by a project. If the project you are looking at from Apache is ‘Incubating’, then it quite rightly should induce a certain sense of doubt. The meme you assume is in place may not yet be in place.
As I said at the beginning - I view this all as an experiment the open source community as a whole is partaking in. I think this stage of the experiment has been successful, and not only are we seeing more and more foundations out there now, as groups seek to put the ownership of their code out of the hands of a single free entity (such as an individual or a company) and into the hands of an entity whose hands are far more tied, but that we’ll also see more common-meme sharing enterprises starting to rise. I think both Spring and JBoss are likely to continue their growth as major (Java) open source enterprises.
When you look away from Java, I don’t think you see this as much. I think that might come down to the open source Java community being closer to the typical enterprise companies than many other parts of the open source community. The shared-brand meme has crossed from the enterprise companies over to the open source Java community and we’re increasingly not interested in anarchy.
To answer the first paragraph, which I’ve probably wandered from and not done a good job in reflecting on. A lot of the arrogance/stuck-upness is due to our memes, non-competing sound like arrogance (I’m sure there are other memes from the list that communicate poorly), crap quality comes from people outside Apache being annoyed that the shared-meme attracts users to a product that is worse than theirs, and absorb the world is obvious. The shared-brand is working. People like our values.
May 15th, 2007 at 9:44 am
Apache is by far, in terms of its community, the worst example of this sort of endeavour. It has fewer participants than most, and has the most arrogant self-absorbed approach.
It’s kind of interesting actually, the ignorance shown by many Apache members. They honestly do feel that they have a ’special’ community that is more open/transparent/merit based than others, when it’s the exact opposite, and is by far the most ‘little boys club’ish org of them all.
Honestly, it really does illicit the sort of snickers common when MS proclaims how they ‘get’ linux or open source. They’re not lying, their ignorance and shallow world view really does buy into that. Apache likewise is a small insular community, and Ted is a great example of someone who honesty believes that it’s special and different to any other open source forum through sheer ignorance and ensuring that he only ever remains exposed to his small band of stuck up gits.
Apache provides value, that’s plain for all to see. Apache is also a superb brand, but that honestly does seem to be a matter of sheer luck and ‘right place right time’ than any ongoing achievements or activity.
May 15th, 2007 at 4:30 pm
Can we have an example or two of “arrogance”? I’ve often heard that quoted but I’ve never seen the organisation been very arrogant as a whole (same might not be said of individuals).
May 15th, 2007 at 5:27 pm
I’m more of the mind these days that it doesn’t matter much where the project is hosted, what matters is that the people involved are actually participating and collaborating.
May 17th, 2007 at 9:44 am
@Hani: In such a large group of people there are good, bad and all variations in between. So doubtless you’re right about some, but IMO generalizations about groups suck. I also think that the primary communication tool (i.e. mail) is v.poor since it doesn’t reveal well the attitude of the sender leaving people to make their own (incorrect?) assumptions - people I’ve met that across badly on lists are often great face-to-face. Another aspect is that since mail lacks a personal interaction its often the case that people say things that they wouldn’t or would put differently if it was in person. I was shocked to find that when meeting you in person Hani you seemed like a nice guy - which isn’t reflected in your blog
On the ‘little boys club’ish org point - although the ASF says its on merit, I would say its more about playing the game. It takes patience and a certain attitude to become part of the ASF. You need to maintain a calm, polite attitude whatever frustration you feel. For example, having a contribution ignored (or despite the certainty that you’re right) being told you’re wrong or talked to in a condescending way can make people what to scream F**K YOU!. But doing so is pretty much a guarantee that from then on you’ll be shunned. I don’t doubt that the feeling of having to suck up to those who have the power isn’t, in part, a cause for the sentiments Hani expresses. Especially if someone doesn’t consider the software/project/person in power that good. I’m in two minds about this - people should be able to express frustration without consequence but at the same it would be bad news if mailing lists became hostile environments. For those that do manage to get into the ASF the danger then is that it feeds the ego and inflates the sense of superiority. If that does account for some of what Hani thinks its still a disservice to apply it to a whole group since there are those (more, perhaps less vocal?) that keep a sense of proportion.
@Henri: Fortunately I work for a small company and get to chose what software to use without considering the brand. In previous life I was in the corporate world and the bureaucratic nightmare of changing anything was horrendous - esp in evaluating/approving new software. Apache’s brand success is in having gained corporate acceptance which since Apache as an organization has gained approval - bringing in a new piece of Apache software is far easier than from an entity/person that hasn’t yet achieved that.
I put that approval down to two things: The Apache Web Server and “policy and processes”. The success of the web server caused corporations to consider the ASF and when they look/looked at the ASF the “policy and processes” fits well with their CYA corporate bureaucracy and gives them something to point to to justify their approval (e.g. the license, project operation, releases, IP (e.g. CLAs), Source Files, 3rd Party Licenses, Crypto, etc. etc). I wouldn’t put too much emphasis on the license - since anyone can use the Apache License without being part of Apache. The difference is that Joe Bloggs on sourceforge may use the license, but how do big corporations know, for example, that what’s being released hasn’t been copied from an un-authorised source. With Apache they see that there is policy and process in place which tries to prevent that from happening. Which means if something goes wrong they have somewhere to point to justify their original decision.
These policies are IMO valid reasons for corporations accepting ASF software. However I suspect that some assume more that are not valid. For example, that ASF software is higher quality or better than an alternative non-ASF work or that the project is more likely to continue/succeed. They are probably not aware of how close some ASF projects are to being un-releasable. The three +1 votes for a release rule is good until the number of people interested in voting drops to below 3 - then the projects in trouble. Also the ASF doesn’t do “maintenance” mode well and there isn’t any realistic path for re-vitalising a project, even if there is community interest - since it needs three +1s to bring in new people. Although the policy and process works well in attracting corporations to the ASF, it does hamper bringing in new blood and innovation and the idea that everyone has equal power means that idiots like me have as much say as the talented developers and I suspect that the benevolent dictator is a better model for successful projects.
To be honest I don’t think there is a solution for the type of criticisms the ASF receives - since the policy and process which satisfies corporations and drives its brand success limits the freeness and openness (from a people joining and taking part PoV) of the ASF as an organisation. I guess the question for each person is do I play the Apache Game for the visibility it provides - or go elsewhere?
May 17th, 2007 at 4:52 pm
Leo: No company is arrogant ‘as a whole’, JBoss as an entity is not arrogant, but many of its leading figures are. The same applies to Apache, the official statements are all well and good, but it’s the individuals who come across badly (and yes, the loud obnoxious minority makes the probably large silent well behaved majority look bad).
This is something that no other open source repository suffers from, so it’s a useful data point. As for examples, you don’t have to go far. There are many many threads on the incubator and other lists that show that many apache people are, uhm, I honestly can’t think of a better way to put this…stuck up their own asses.
Niall: You’re right, to work in Apache, you have to play the Apache game (as distasteful as it is to everyone) for the visibility. This to me is NOT a healthy model for an open source community. Apache is a lifetime membership, all you have to do to get in is do one good thing, and you get to feel smug and obnoxious about it for all eternity without ever having to give back and earn your keep. How is that a meritocracy?
May 20th, 2007 at 3:48 am
Henri, there is a connection that I was thinking about between this and another topic I have often thought about: financial bubbles. In a financial bubble, there is some class of things that gets extremely overvalued. One (minor) aspect of this is that it invariably spawns a certain kind of writing, where commentators try to explain, via quite forced, tenuous arguments, why the valuations (that seems so crazy) actually make sense. You know, at the peak of the internet stock bubble, there were people trying to explain why the prices of internet stocks actually
made sense (except they didn’t). Or more recently, people trying to explain why certain real estate prices that seem crazy actually aren’t crazy… (except they are….) In that vein, it’s like you’re making a somewhat half-hearted effort to explain why the overvaluing of the Apache brand (which you yourself recognize, I think) makes some kind of sense. (Except that, again, it doesn’t…. :-))
Niall (and Henri), the reasons that corporations might prefer ASF branded OSS software over higher quality alternatives, are, on serious analysis, pretty thin. Even the business-friendly ASL license is not a major consideration in most scenarios. Most corporate users are interested in free software in the sense that it is free as in beer. The various software licenses are largely concerned with modifications, derived products, and whether those in turn must be open sourced. When you use a software product as-is and never make any changes, all of that is moot, and pretty much any of the commonly used licenses are okay. That Hibernate, for example, is under the LGPL rather than ASL, is of no concern to somebody who just wants to use the library as-is.
The legal CYA aspects, that ASF has somehow done due dilligence to check out the IP issues, does not stand up to that much serious scrutiny. I have been (at least de facto, it’s not official…) project lead of FreeMarker for 5 years or so. FreeMarker is a core component in at least 2 Apache Projects, Struts 2 and Open for Business. When you use either of those things, you are necessarily using FreeMarker. I can tell you that nobody from ASF ever contacted me or our community to verify anything about the IP. AFAIK, their verification amounted to looking at the web site and taking our word for it; this is exactly what a corporate user, using FreeMarker directly, not as part of Struts or OFBiz, would do. So what is ASF providing on that legal front? I, frankly, cannot take the whole thing that seriously.
Of course, the corporate types may believe that there is greater dilligence on that legal front, but that just gets us back to the bubble issue, people valuing something much more than it is inherently worth.
Most of the bubble phenomenon (and it’s not so dissimilar to the actual financial bubble) IMO resides on a herding phenomenon, that people perceive safety in numbers. Everybody else is doing something…. To use something like Struts or something that is considered sort of the standard thing to use is perceived as the safe choice. The decision making in these companies is inherently conservative and the decision maker can reason that if he opts for Struts or something from Apache (everybody has heard of Apache Web Server) then it is the path of least resistance. Even if things go horribly wrong afterwards, people will likely not criticize the technical decision to use the ASF stuff, since it’s stuff that everybody else uses, etcetera.
As for the issue of ASF culture, I have two basic comments to make about that. My own experience backs up that of Hani; the culture in ASF communities is frequently just terrible and that people tolerate it there shows that many of the people have no broader experience of other OSS communities, so they don’t realize how bad it is. I would venture to say that most corporate users of the software do not have much of a sense of the Apache culture; I doubt that this is a factor causing them to use the products. In fact, I would add that anybody who was really familiar with ASF culture and had experienced other OSS communities, would, on the contrary, be rather put off by it.
However, all of this is IMO a rather inevitable consequence of the “bubble”. Consider what the world is like for somebody who starts a project on sourceforge.net or some place, and announces it on freshmeat and a few other places. When (and it doesn’t necessarily even happen) somebody shows some interest in your work, asks some questions, and so on, it’s like gold. You’re dying to get some feedback on this stuff that you put so much work into. Of course, the project may eventually get a lot bigger, but those are the origins and, for people not working under an umbrella like ASF, your baseline assumption is that users (and even collaborators) are not that easy to come by. To treat somebody contemptuously who actually shows interest in doing some work is, for me, unthinkable. Yet this is very common in Apache land.
Now, consider the world for somebody in on ASF. What you do is Apache Foo, or Apache Bar or Baz, and it gets a huge amount of attention and usage, almost completely independently of its quality. Well, this can go to your head. But compared to the scenario outlined before, where is it more likely that users will be treated contemptuously?
Though it’s annoying, I think if you’re going to analyze the phenomenon, you see that it’s just a natural consequence of the overvaluing of the brand.
As a final point, directed at Niall, I am very skeptical of your idea that ASF has to work the way it does to be appealing to corporations. For whatever reasons, it does work the way it does, and for whatever reasons, it does have huge corporate acceptance. But that does not mean that there is much actual causality between those 2 things. I don’t think that most corporations that use ASF software have much of a sense of the culture there or how things work. For example, most of them probably are not even aware of all this silly “Apache Way” rhetoric. I don’t think all that stuff is necessary to attract corporate users, because anybody with half a brain who looks at that, would realize that it truly is nonsense.
I honestly don’t know where most of it comes from. For example, all this arbitrary stuff about voting makes fairly little sense. And, as Niall points out, if you need 3 votes, what do you do if there are only 1 or 2 people around? It’s like saying that any medical procedure requires a second opinion, so if there’s one doctor in the room and some guy is dying, you just let them die rather than let the doctor do something….. (there’s only one doctor and you can’t get a second opinion…)
And, I read on some page about the Apache Way that in the Apache Way, there is no concept of a leader. Where does this come from? From your comments about a benevolent dictator model, Niall, I see that you don’t really take that too seriously, right?
As a final, final point about this, I note that Wiley and Sons published a book entitled The Apache Way in late 2004 and it is now out of print. Amazon sales rank 4 million and something.
I do not think that corporate adoption of Apache Software has that much to do with all the various bureaucratic procedures of ASF and the so-called “Apache Way”. The corporations in question do not have much of a sense of how ASF is run — nor, judging from sales of that book, are they really that interested.
So the basic argument that ASF needs to saddle itself with all this bureaucratic stuff and this mostly ludicrous “Apache Way” nonsense in order to attract corporate usage does not really ring true to me. Frankly, the Apache Way stuff seems to correspond to some other deep psychological need, maybe somewhat similar to the need of pigs to wallow in their own excrement. (Except you replace the pigs in the above by bullshit artists…)