Gearmand 1.1.9 Released

The basics:

• Added gearman_task_is_finished()
• Improved SSL support.
• Exceptions are now supported.
• gearmand excepts its root CA via the environmental variable GEARMAND_PORT. 
• libgearman will now except GEARMAND_CA_CERTIFICATE, GEARMAN_CLIENT_PEM, and GEARMAN_CLIENT_KEY

http://launchpad.net/gearmand

From the feedback I am getting the SSL appears to be working.  If you are interested in learning about how the SSL works, you can find out more about it on the docs website.

I am curious to see how many people get tripped up over the fact that an SSL setup requires that you not use self signed certificates (I may support these later, but I wanted to get it right in the beginning).

Unless something major comes up the next version will be a release candidate and the version after will be the formal release :)

MySQL vs NoSQL vs Postgres vs SQL Server, Cage Match

Tonight I went to compare the usual suspects for a set of slides I was doing and thought I would share the results.  

I do not believe anyone should be surprised by the results. MySQL became less interesting once it was acquired, and the number of Postgres deployments is greater than what all of the NoSQL market combined adds up to. When looking at emerging technologies, press time given to a single software product has no relation to what its actual market share is (which isn't as obvious as one might think to everyone).

When I get asked what my current opinion about NoSQL vs SQL is, I say that I continue to see NoSQL as a spotlight highlighting the failures in the design of the current relational database software vendors(*). Which should not damn the concept of the relational database, but somehow it happens anyways.

The two biggest issues that stand out to me are that current leading relational databases never solved scale out very well, and online operations are too expensive.

Would a document model really matter if schemas could be altered online? You would still have the weak vs strongly typed arguments that can exist in schema/table design, but if the operation to add additional characteristics to a table (object) was online or automatic I believe a lot of the arguments you see for schema less design would go away.

The other giant failure was that the vendors didn't approach scale out as a strong requirement for future designs. Going back a few years you can see where the vendors would talk about shared nothing designs, but only in the scope of support handful of nodes in a cluster. A hundred or so at best, which is small potatoes for where some of the NoSQL vendors who focus on scale out shoot for.

Its funny to think about how much has been written on this topic that has focused on the SQL language, when no new language has emerged from the NoSQL market that has any size-able adoption. At this point most of the NoSQL vendors now talk about how they have, or are, adding a SQL interface to their products.

The innovation hasn't been in the language, but in the design of database engines themselves.

The other bit of humor in this, is that if you take apart how most of the major vendors have implemented their storage engines, you would find that in the inner design few are strongly typed, and few track relationships in the core of their design. Most of the major vendors could do more operational changes online if they were willing to put reasonable scoping on what can be altered. The Innodb row format is inflexible, but the addition of some version information and a catalog documenting the versions would fix that.

To sum this up, the need to preserve backwards compatibility is what has kept the relational database vendors from solving the needs that caused the NoSQL solutions to come to market. I don't see any of them tackling this anytime soon; the margins on their current products doesn't dictate the need for them to do this.

A final thought about the state of the current nosql databases.

One of the "hard things" that almost(?) none of the current NoSQL vendors has tackled is the JOIN, i.e. the fundamental feature which is required to do anything beyond simple key/pair or basic search. Join optimizers are hard to write(*).  Network aware join optimizers are particularly difficult to write mainly because direct attached storage, AKA a hard drive, is a lot more reliable on a second to second basis than the network.

*) I also get asked which one I would pick if I had to pick one. Currently I suggest Cassandra, though more often then not a table with a key and a blob would work as well.

*) One of the nice things about MySQL's design was that someone designing a new engine could focus on the datastore and punt on the rest (well, sort of,... the pluggable storage engine interface never matured to the point of really playing nicely with other people's designs).

Pioneer 1021 / 1121, Airplay, You suck

After a year of owning both a Pioneer VSX-1021 and a VSX-1121, I would never ever buy Pioneer equipment again.

The Airplay software on the Pioneer crashes a lot. When it doesn't crash, it plays music out of sync with other Apple airplay devices. 

The HMG server? Who let that ship? Did they look up from their Atari 2600 for more then a few seconds before saying "Menus! What sort of crazy new concepts for an interface will you think of next? Ship it!"

Their support for their iPad and iPhone application has been non-existant.

On the other hand, the Pioneer sounds better then my 20 year old Onkyo amp (I think). 

Someday, someone, is going to build a modern amplifier right. 

I really doubt it will be Pioneer.

You can be right, or you decide to save everyone else a bunch of time...

This is one of my favorite valgrind finds at the moment (well, last few months):

|| ==881== Syscall param sendmsg(mmsg[0].msg_hdr) points to uninitialised byte(s)

|| ==881==    at 0x386BAF3D6B: sendmmsg (sendmmsg.c:36)

|| ==881==    by 0x386DE0B4D6: __libc_res_nsend (res_send.c:1132)

|| ==881==    by 0x386DE08BDF: __libc_res_nquery (res_query.c:226)

|| ==881==    by 0x386DE097B9: __libc_res_nsearch (res_query.c:582)

|| ==881==    by 0x5E8CAC7: _nss_dns_gethostbyname4_r (dns-host.c:309)

|| ==881==    by 0x386BAD9AF1: gaih_inet (getaddrinfo.c:845)

|| ==881==    by 0x386BADD750: getaddrinfo (getaddrinfo.c:2359)

If you really want to be right, and not cost a lot of other people time when you know you are right and valgrind is wrong. You could leave a snarky comment in the code where you create a fix to silence the valgrind error.

Or you could decide to not fix the warning, and cost a lot of other developers their time as they trip over you academically correct code which spews errors when run through valgrind.

Child Processes

When we, some of my peers, talk about "child processes"...

20 years ago the discussion was around the question of "who all is running netrek, and can we have someone who is root kill their process?". This was a quality of life issue, a very real first world problem.

15 years ago it was about zombies. Software usage was such that we started noticing quality. Child processes that became zombies. Seriously, zombies.

10 years ago  it was all about tuning Apache such that it had the optimal number of "children" such that you would not go into swap because you miscalculated the amount of memory that would be used. We often had too many children.

5 years ago the debate around child processes vs threads became a real conversation due to the changes in hardware. 

Today? 

I find my peers talking about child processes and sleep time, where child processes are in the 1 week to 12 month range with a average sleep time that is anywhere between 2 and 8 hours.

*) Can you find reversed joke in the above statement?

Installing Open Stack

It took me 4 hours to go from bare metal to a small cluster of computers today.

The install was done on Fedora (Redhat), and included Horizon, Nova, Swift, Cinder, Glance, and Keystone. This was using Folsom.

I hit exactly one install bug that I fixed in under 10 minutes. I fed Pepper once. 

Not bad.

Burning Man, Steve Jobs, The Importance of "use" over "testing"

My only real contribution to Burning Man, beyond the price of my ticket this year, was to help Ignite BRC by carrying a monitor to Center Camp, and to point out that Steve Jobs had now left the building.

How did I know that Steve Jobs had left the building?

I did the see the news reports, the book, and had walked past the memorial that someone had built on a local street corner. This wasn't like the moment in grad school where I had to ask "O.J. did what?" when one of my students tuned on a radio in class to hear the jury report. 

But how did I really know that he was dead?

A week before Burning Man my laptop failed to work with a projector. This is how I knew Steve Jobs had departed this world.

I doubt Steve ever touched Mountain Lion.  My presentation was given from a brand new laptop, with a shiny from the factory setup of OSX. 

 

Out of the box my OSX setup did not work with the projector. 

That would have never happened if Steve had been actively using Mountain Lion. 

Can you imagine what would have happened to the developers if Steve would have been alive, let alone experienced, such a simple failure of technology? 

It would not have been pretty.

One week after my own experience with OSX not working with a projector I found myself in a different environment reliving project-fail. 

There is something to be said about knowing for certain that the reason that the laptop sitting in front of you is not able to run a presentation is because the OS is at fault.

I was standing in the desert, in a giant circus tent, with dust in the air, staring at a hodgepodge of cables that were connected to some ancient CRT. There is a lot of things that might be contributing to the problem.  

When I was asked, "why isn't this working?"

The most useful thing I could contribute was:

"Running the presentations off a mac isn't going to work; the laptop is running Mountain Lion. Go find a Windows laptop."

The presentation was moved off the Mac, and onto a PC. 

The PC immediately worked. 

I am wondering if anyone at Apple used Mountain Lion to give a presentation before it was released. It is hard to imagine that someone didn't run into this problem long before Mountain Lion was released (as far as I know Apple has not  fix the bug yet).

I am sure that there is a lot of testing happening each release.

Testing software is not the same as "using" software.

Apple is a company, someone, at least once a day, must be giving a presentation. 

How did they miss this?

Steve Jobs would have presented Mountain Lion to the world using Mountain Lion. 

If that presentation had blown up on him,...

Egads.

Drizzle 7.2.3

From the release notes:

Fix for CTRL-Z for shutdowns. Many updates for JSON server. Improvements completed on Catalog support. ZeroMQ and Gearman supported updated. Updates AUTH_HTTP authentication module. Documentation enhancements. Testing framework updated. Regex Policy updates.

Stewart can say a lot more about Catalog support, but this is one of the cooler bits of work that is happening ongoing. The original work was began for Rackspace and continues today. Virtualization inside of the database is long overdue.

The JSON server is another one of the locations where we see a lot of effort. Being able to support JSON queries directly seems to resonate with a lot of folks today.

For the record I wrote zero lines of codes in this release. Work continues, and in the case of Drizzle it is all about work coming in from the community :)

Drizzle 7.1 Released

Drizzle 7.1 was released!

The laundry list for features and improvements are pretty long, so I will leave that to the main blog announcement.

The part that I find the most interesting? 7.1 was the effort of a lot of developers and people who do the heavy lifting of promoting the project (and the efforts around a project like this are huge). It takes the resources of a half a dozen companies directly, and a number of companies indirectly.

Compared to 7.0, Drizzle was done by volunteers, not paid Drizzle employees, who find the project fun and interesting. These are people who want to be involved with the Open Source Community, and who find value in the work (and looking at our incoming Google Summer of Code applicants there are more people to come).

The biggest surprise that some might fine? Much of the 7.1 effort was done by new developers. These are people who never worked at MySQL, Sun, or Oracle.

Contributions by new individuals make up the bulk of the 7.1 effort, and I am very happy to see that. It it is evidence of the ongoing growth of the Drizzle community.

One Fat Bird

Opscode's Chef, MySQL, Best Practices

If Chef manages a CNF file, please have it put a comment in the top of the file that it is managed by Chef. Do not assume that everyone will believe that every file is being managed by Chef. In general, you should have Chef leave a comment in every file that it manages (and someone at Opscode should make this a default feature in Chef).

Do not have Chef reboot the database. Databases are designed to run for years at a time. Many parameters can be set while the databases is running in such a way that it does not need to be bounced in order to make the parameter work. There are exceptions to this in non-production environments.

Need to change the schema? Do not have the Chef create a table, and then do alter table after alter table to install a new system. This is very painful to watch.

* Thanks to Alex Howells for the idea of always putting a comment in the top of the Chef script.

Patrick!

MySQL Conference, Percona, the Ecosystem...

This year O'Reilly isn't running the MySQL conference, Percona is doing it. Santa Clara, April 10-12. The usual time and place.

This is great news. They run a great conference. They always get rave reviews.

This isn't the first one they've done. They've done San Francisco, New York, London, more.

What's different about Santa Clara?

This isn't their typical 300-person one-day conference. They're picking up the annual MySQL users conference and carrying the torch forward. This is a 1000+ person, huge expo hall, 8 concurrent tracks, several days conference.

This year is a banner year for the MySQL ecosystem.  Why?

Because now the conference is focused on MySQL again. This is the central conference for the ecosystem. And now it's managed by a company that understands both the community and business side.


And the MySQL Ecosystem? Thanks to cloud vendors it continues to grow.
The sessions? Better than ever. In the past there was a lot of auxiliary content. Take a look. This year it's easily the best technical content I've ever seen. And I've led the selection committee for years.

The sponsors and exhibitors? Pretty much anyone who's anyone will be there.  Looking for solutions to your MySQL problems? You'll meet the folks who can help you. Visit the expo hall. There is great sponsor support for this year's event. HPCloud, Facebook, Clustrix, Google, et cetera.

The keynotes are nothing to sneeze at too. There will be talks from Marten Mickos, Mark Callaghan, myself, and more of the old gang.

Is there more? Yes there is.

There are the community awards, BOFs, lightning talks. There is a Tuesday welcome reception, Wednesday community networking.

More? Yes. There is not one, not two, but three follow-on events. Stay an extra day and on Friday you can attend Drizzle Day, SkySQL and MariaDB Day, or Sphinx Search Day.  Three awesome technologies for MySQL users.

This is a really big deal. If you are even slightly interested in MySQL, this is the event of the year. Don't miss it. By the way, early-bird pricing is almost expired. Register before March 12th or you'll pay more for the ticket and the hotel.

Drizzle Day at the Percona MySQL Conference

The major annual event in both the MySQL and Drizzle worlds is the conference week in April. This year Percona is organizing the Percona Live MySQL Conference and Expo from Apr 10-12 and as usual we follow up with a Drizzle Day on Apr 13. It's the 4th Drizzle Day!

I will be giving a keynote at the main conference, on Wednesday April 11. I will be giving a keynote on the state of the MySQL Ecosystem and how cloud is evolving it. (HPCloud is a sponsor of the conference.) Of course, it is about Databases in the Cloud so MySQL and Drizzle are still with me :-)

There are a few Drizzle specific talks though. Check out Scripting MySQL with Lua and libdrizzle inside Nginx and  Getting Started with Drizzle 7.1.

Then on Friday, April 13 it's all about Drizzle. I will be giving a mini-keynote to start the day, and then we have great talks about new and old features in Drizzle. The Drizzle Day is free entry, so even if you're not attending the main conference, but if you're interested to learn about using Drizzle, please consider joining us. At the end of the day there is also some content for those who might want to start hacking on Drizzle itself. Especially if you live in the Bay area, just pop in, it would be great to meet Drizzle users and hackers in person.

Oh, and I also want to say it is really great that Percona and Technocation are sponsoring this event and also that SkySQL has invited us to their lunch and after party. Since Drizzle is purely a community project, we really appreciate this. It is what makes Drizzle Day happen.

Speaking of lunch: Please RSVP to us so we know how many are coming.

Error Messages

I have written a number of libraries that are frequently used, and I have yet to find a pattern for error messages that I am completely happy with.

Let me tell you about a few of my thoughts on this.

I have found that there are two parts to an error, the code and the message. The code is a numerical value, and the message is an expression that you expect a human to read.

Error codes and messages do not have to map one for one.

You only really want to provide a specific error code if you believe the developer who is working with your code can do something about the error.

I've definitely gotten myself into the trap of creating a dozen or so error codes that all relate to how a host has failed a connection. In almost all cases of a connection failure there is very little that the end user of the end user of the library can do.

While one error code might be fine, you should create specific error messages which provide more information for an end user to diagnose a problem.

The developer who is looking at the error will appreciate the message. It might make the difference between someone spending five minutes, instead of five hours diagnosing the problem.

Never return an error message that is just a number.

No one wants to try to figure out what "error 13" means. A number tells me nothing, and while I could google the number, that is an extra step I don't want to have to take each time I look at a problem.

If you need a paragraph to explain the error, make the error searchable.

If a developer wants more information on the error, they will search for it. Make this a very simple process. It drives your users to your website. This may sound obvious, but I continue to meet people who haven't realized this.

Do not map to ERRNO.

Mapping to ERRNO. ERRNO is not very flexible, and it comes with the baggage of a preconceived notion of what the error means (which does not map across operating systems).

Give yourself enough information to diagnose a problem for the end user.

In the last few years I have taken to embedding the line of the code, and the file that the error was in, every error messages. This has allowed me to better support software that I write by creating context for me. An error is not only a tool for the end user, it is a tool for you to provide support.

Consider the case that multiple failures may occur.

Sometimes you don't have one failure in a given context, so try to store up all of the errors that occur. If you can, design your error system so that you store all of what failed.

In the end,...

Be consistent. Format your error codes and messages in a consistent manner. Make sure that if someone wants to, they can parse them in batch. Never under estime the creativity of an end user armed with a regular expression.

CenturyLink, Suck, US Tech Support

I wake up and discover the internet connection is down to the house.

I log into my router and run the diagnostics software that is built into it. Everything is good (BTW this is an actual router which bridges the DSL signal to ethernet (built by Cisco)).

So I call tech support. The fellow immediately blames the device and tells me that they will send out a new one on Tuesday. It will have built in WiFI, and since "I don't pay for a static IP address" it will be "easy to configure".

Ugh,... I have a routable subnet, and no I don't want your cheep assed device that won't do what my current device does.

We go around and around for  a bit, and finally he gets me his manager.

His manager stumbles through the conversation for a bit, and repeats the "we can be out by Tuesday, and our new device is better...".

I escalate, and he pushes back.

So? I escalate again, and point out that I am happy to call the city and bitch about their service (I also drop words like "911 service", etc...).

I get transferred to Idaho.

This nice woman bounces the line for me, and....

Everything works.

In the future, I am going ask to be transferred to the US first.

Twisted Circus,Going all our for Halloween in Capital Hill

Contributor Agreements, Open Stack's Contributor

I have not been a fan of the Open Source Contributor agreements that have been used for the last two decades. All of them tend to have an onerous bit whereby a developer has to hand over the copyright to a group, organization, or company.

I’ve never felt like this is a very honest exchange. With MySQL you were handing over copyright to a company that was making money off your work. With the FSF I have always been bothered by Richard’s insistence that FSF has the right then to take that code and relicense it. His stance on dual licensing under commercial licenses is my issue with this.

The Open Stack contributor’s agreement is a bit different then much of what we have seen thus far. It basically states, and please keep in your mind firmly that I am not a lawyer, that you have the right to submit the code you are submitting. It states directly that you can do whatever you want in the future with the code you wrote.

It is not Apache specific. I don’t see any reason why it couldn’t be used with a GPL project as well.

It is worded such that the group that the code is contributed too couldn’t just take your code, and then hold the contribution hostage. By hostage I mean that the contribution would sit in a limbo where you couldn’t do anything with it, and would therefor be at the mercy of the new owner of the code publishing it.

The GPL, BSD, Apache, MIT, and other licenses where a watershed in their time. Lawyers learned the licenses, and each of these licenses have been “debugged”.

Every time a company sees a new license, or a new legal agreement, there is a huge bar that must be met before it can be signed.

If you are an engineer, think of lawyers as a picky c++ compiler. Some lawyers issue better warnings then others. Some organizations turn the compiler flag “all warnings to error” on. Other organizatiosn not only do this, but add they add in -Wextra for good measure.

We have software licenses, the OSI stamped out quite a few of these.

What is missing then?

• Agreements for developers who are on advisory boards. 


• Contractor agreements that carve out open source projects such that they don’t become entangled with “work for hire”. 


• Contribution agreements.

Take a look at the Open Stack contributor agreement. Could it be used as a standard for open source contributions?

Where is the advantage in using it? We already have a long list of companies who signed it for Open Stack. It has been debugged, and a number of large companies are willing to sign it.

DSC_0262

Spoon full of Sugar, Oracle and the Open Core Model

From the 451 Group:
“MySQL flirted with the open core licensing model in early 2008 with plans to introduce new features into Enterprise Edition that would not be available under an open source license.”

MySQL didn’t flirt with, it was going to do it. 

Why? Because we were asking the question, “how do we pull in customers to make more money”. 

MySQL was going to put the new backup API, which never materialized, into an Enterprise branch. 

It was a lousy idea for the following reasons:

1) There was no internal API in the server for this, so the engineering was going to be messy and expensive. 

2) We didn’t own the technology that was needed to even do this (Oracle owned Hot Backup)

3) Percona has an awesome tool for doing this, that is Open Source (http://www.percona.com/software/percona-xtrabackup/)

4) Backup is a core feature everyone needs, and some of those “everyones” are the folks who manufacture tools that you want to have work with your product.

5) When we were going to announce it, we hadn’t even written it/completed it. It was vaporware. 

It would have been a horrible move, and would have caused Chaos for no particular reason. It was dead on arrival, and when it was to be announced as a strategy since it didn’t even exist. 

Lets look at Oracle’s move. Both the authentication module, and the Thread Pool come into the MySQL server as plugins. If the engineering of the MySQL server continues in the current direction (which is somewhat flattering to Drizzle I might add), then they are on a good path (if I can find my blog entry where I talked about this as a good strategy, I’ll link back to it here). 

Much of the hubbub around Open Source, Community, etc, in regards to this are a bit inflated I feel. They haven’t touched the core product, and they are creating API. Are they possibly hurting themselves in regards to ubiquity?

Doubtful. 

Would I pick those two pieces? No, but they aren’t the last two I would pick either.  If Sun had continued as a company? Something similar to this would have been done as well.

From an engineering and usage stand point?

The first person who sniffs at the authentication mechanism who knows anything about security is going to freak.

The Thread Pool can only be used by a very limited number of users (and there are some restrictions on what can be done in the server while it is in use). MySQL’s IO was never designed for the Thread Pool, and there is a lot of engineering work that would need to be done to make it work. 

Still? People will use both, and I am betting some customers will want them badly enough to pay. 

If they are really badly needed? Well then someone will write an open source version of both.

I have no great love of Oracle, but this is really not a big deal at all. The original GPL’ing of the Public Domain/LGPL clients was a much bigger deal.