Big “Fine” Arrow

MySQL, State of the Ecosystem 2011

A number of years ago I coined the term “the mysql ecosystem”. I did it at the time to express a view that MySQL had moved beyond being just what MySQL AB defined “MySQL” as being. 

 
It was a radical thought at the time. In part because when I expressed it, I did it not only outwardly to the world, but inwardly to the company as well. Many at the time thought that the ecosystem danced at the whim of the MySQL AB entity. When Peter Zaitsev left to form Percona I remember very clearly a management meeting where there was a hubris that his business would amount to nothing, and that he was missing his opportunity to be a part of something greater. History is of course writing a very different story.

So how is the ecosystem?

It turns out it is pretty healthy.

I wasn’t sure if that was the case up until a couple of weeks ago. I was having lunch with Moshe Shadmon of ScaleDB and I asked him “Do you think the market is collapsing?” 

His response to me was one of enthusiasm. He pointed out to me the obvious indicators. The growth Amazon has created with its relational database service, and the continued growth in applications that support the MySQL interface.  

The conversation put me into a really positive mind set about the community. What did I find at the O’Reilly MySQL Conference?

I found a lot of happy people. I saw adoption numbers which show positive growth.

What didn’t I find? The overwhelming negativism of the previous two years that I have sensed in the community was not to be found. It has at times made me question not only my involvement, but the involvement of Drizzle* in the ecosystem. I personally don’t wake up everyday wanting to welcome that into my life. 

But what was the vibe of the community this year, and that of the conference?

This year the negative vibe was seen as something that was not only as ugly, but as something that was an aberration. An evolutionary path that the ecosystem seems to not be taking. That is pretty awesome.

What are the big questions facing the Ecosystem?

Oracle. I watch the MySQL trees, and I see that they are having an overall positive influence on the codebase. They are making good decisions, none of which appear to be malicious in nature. I hear from people who are using it, and I get an overall positive view of the work. 

The people I ask? 

They aren’t the shills that are trying to gain favor with Oracle, these are people who have 24x7 needs who don’t have the time to write blog entries, and who see MySQL has just one piece of their overall architecture. 

If you are using MySQL today, and you need a solid path forward on it as a platform?

I’d stick with what Oracle is creating.

Oracle will be Oracle though. They have a giant marketing machine that will not want, and by policy not allow, events to occur which favor a product like MySQL over other products. Oracle Open World will not be a MySQL conference. MySQL will be a track in that conference, a booth at best. Oracle will push for venues that they control. Oracle will push for users to adopt their stack, and MySQL is just another cog in their system. A vector to attack Microsoft? A product to keep at bay the growth of an open source database? 

It might be all of that and more, but it will not be a crown jewel. The company is too large to focus its attention on MySQL, and the money that it obtains from MySQL is not enough for it to ever take center stage. 

In the end?

The attention span of large companies is quite small, and at some point it will fade.

Will Oracle have a MySQL Sunday again this year at Oracle World? If it does, will it have one the year after?

There is nothing wrong with this, it is just the nature of large companies.

Percona.  Percona is impressive. They do excellent work based on an excellent reputation which they have grown by doing the right thing. I’ve been asked before if they will become the next MySQL. I don’t believe they will. Percona looks to be the next Electronic Data Services.  

Do they have a server product? Yes. Will Percona Server be the next MySQL server? No. Is that because it is inferior? No. It is because Percona server is about delivering on their ability to be the best at MySQL consulting. It is not going to go away, but I will be surprised if Percona decides that it is their one and only product that they service. Percona Server is an asset for them, but they show no evidence of being singularly focused by their own product. 

SkySQL. SkySQL has a great feeling to it. It has the exciting feel that MySQL once had, but I see no signs of the baggage that MySQL AB gained in later years. The people they are hiring are excellent. In the MySQL world they could very easily take the dominate position in the next year. 

Monty Program. I don’t feel like I can really say much here, but I don’t want to say anything by leaving it off the list either.

Amazon. They were a sponsor of the conference this year. They are certainly a player in the ecosystem, though for the most part a silent one. From an engineering stand point I believe they have one hell of a challenge. How do they continue to provide MySQL services without a deep technical bench and a roadmap that will allow them to adopt new versions of MySQL? They don’t shape the MySQL universe in the ways that others do. They do not provide code, and they do not influence the direction of the product in any manner that allows them to influence beyond the scope of their own service. 

Their service though? Amazon could be setting a stage where we see the MySQL interface solidified. If a large portion of MySQL apps are shaped by the question “will this app work in the Amazon cloud?” then they will have their say.

Are there others? There are plenty of others. Canonical and Redhat will shape the Linux distributions, and that in turn will shape what users first see. There are players like Infobright who will shape the analytical market. 

Postgres continues to make progress. When I ask folks who study the market how they see Postgres I never get a response that it is on their radar. But when I ask operation folks? There I hear about its growth. At some point an application is going to come along that will change the view of the market. 

The MySQL codebase? It is GPL. Nothing has changed about that, and nothing that we are seeing, or that is talked about in private conversation, leads me to believe that is changing. There was some hubbub at the conference about Oracle removing the FLOSS exception from the codebase. There was talk that this created a situation where at any moment Oracle could change the exception and squeeze someone via a license gotcha.

When it was brought up it made me suspicious as well. 

The thing is? Its up on the website still, and the page has been recently updated. It has also been cached and stored by Google. Removing it from the source code doesn’t mean much.  

Its good to be suspicious, but I suspect that all the removal was, was a simple mistake made by a blanket policy about communication. Oracle’s open source behavior, its table manners, are haphazard. I don’t believe you can expect anything else.

In the end?

The MySQL Ecosystem is doing just fine. There are challenges, but there has always been challenges. 

*Drizzle I leave Drizzle out of the discussion because I both feel like it is inappropriate to mention it because of my own involvement, and because I actively debate our involvement in the MySQL Ecosystem. I’d rather push for our own environment.

A reminder...

A reminder...

The new blog can be found at http://blog.krow.net/

The recent DOS attacks against LJ means that my attempts to keep both in sync is failing, so be sure to update to the new link if you are curious…

Libmemcached 0.49 Released

We have released version 0.49 of libmemcached. This version has part one of two of the new configuration work.

You can find an example of it here: http://docs.libmemcached.org/libmemcached_examples.html

There has been a lot of updates to the documentation, http://docs.libmemcached.org/, and there was a big improvement made to the code and we are finding that we better performance for large objects on Linux.

Configuration now is much simpler, and is no longer different on a per language basis (ie once the downstream drivers pick up this version you can configure any language in the same manner). There is native virtual bucket support, and I have a developer assigned to looking through the changes in 1.6 so that we can support it based on server responses.

From talking to Alan (ie Dormando) he has some ideas on how this will work in the server, and I am excited to see what will happen here.

On the microcontroller side we will be publishing a port of the micro client to the ARM STM32 (which follows on the work we did for the ATmega328).

memslap has been split off into the older, very portable, version, and the less portable memaslap.

There is a new error message system which first goes into this version which will give you a lot better info on what is going wrong if something is going wrong.

The pool interface has been updated as well, which means if you don’t want to run a proxy, you don’t need to. This was an idea Trond had a couple of years ago, and we have updated it so that you ran reduce the complexity of your environments.

Or maybe? #bfa

Follow The Arrow!

Additions to the Gearman API

There are a number of new things which are coming to the germane API that I thought I would blog about (and before anyone asks, all of the old API is available as well).

Here is an example of usage:

 gearman_function_st *function= gearman_function_create(gearman_literal_param(WORKER_FUNCTION_NAME));

Functions are now a type. There is a lot being done to make functions more powerful by having context and meaning within the server. This begins to expose some of that. gearman_function_t itself being a type allows us to add characteristics to the function. Functions can also now be written to use any valid UTF-8 as a name, which was something that was not really possible in the past (it worked in almost all cases). 

  gearman_workload_t workload= gearman_workload_make(gearman_literal_param(“test load”));

  gearman_workload_set_background(&workload, true);

This is an example of generating a workload and will be passed into the server. The workload type is very lightweight, it essentially wraps the characteristics of the work. It does zero memory allocation of its own when creating the type to it is simple to generate them and not be concerned about having to clean them up. Things like background, priority, scheduled time, and a few other characteristics can all be specified. Just like function it is not tied to any single client connection so it can be resent to multiple clients without worrying the lifetime of any single client.

  gearman_unique_t unique= gearman_unique_make(gearman_literal_param(“my id”));

Unique keys are also now given their own value. You can continue to have the server generate one as needed, or you can create your own.

  gearman_status_t status= gearman_client_execute(client,

                                                  function,

                                                  &unique,

                                                  &workload);

gearman_client_execute() is now the new workhorse of all methods.  By having characteristics on functions and workload we remove the need to have dozens of commands.

  if (gearman_status_is_successful(status)

{

 gearman_task_st *task= gearman_status_task(status);

}

Status types now give you information on what has occurred with a job. If appropriate, you can also have access to the task that was generated from the execution of a job. All of this is now available in the build trees for Gearman. Have fun with it!

libmemcached configure language

In the next version we are rolling out the new configuration language that we developed a while ago.

 
Configuration is hard, and trying to determine all of the little options can be time consuming. A while ago for a couple of customers we wrote a configuration language and a couple of other utilities to simplify the process.

In the next version of libmemcached we will be rolling out the API for it.  Here is an example:

memcached_parse_configuration(memc, "--DISTRIBUTION=consistent,MD5 --servers=localhost:11221,localhost:11222,localhost:11223,localhost:11224,localhost:11225 --CONNECT_TIMEOUT=456 --NUMBER_OF_REPLICAS=2");

It can either parse via a string you pass to it, or it can read everything from a file.

 There are a number of options to the language like “RESET” which will take a configuration and reset it at the point that the keyword is found. END which will allow you to stop parsing, INCLUDE which will let you include files that will be parsed as well.

 There are a number of other bits to the language as well.

The code is now in lp:libmemcached, I am hoping to publish a new version sometime in the next few days. 

Have fun with it!