Friday 27 March 2015

High-availability options for MySQL

The technologies allowing to build highly-available (HA) MySQL solutions are in constant evolution and they cover very different needs and use cases. In order to help people choose the best HA solution for their needs, we decided, Jay Janssen and I, to publish, on a regular basis (hopefully, this is the first), an update on the most common technologies and their state, with a focus on what type of workloads suite them best. We restricted ourselves to the open source solutions that provide automatic failover. Of course, don’t simply look at the number of Positives/Negatives items, they don’t have the same values. Should you pick any of these technologies, heavy testing is mandatory, HA is never beyond scenario that have been tested.

Percona XtraDB Cluster (PXC)

Percona XtraDB Cluster (PXC) is a version of Percona Server implementing the Galera replication protocol from Codeship.
POSITIVE POINTSNEGATIVE POINTS
  • Almost synchronous replication, very small lag if any
  • Automatic failover
  • At best with small transactions
  • All nodes are writable
  • Very small read after write lag, usually no need to care about
  • Scale reads very well and to some extent, writes
  • New nodes are provisioned automatically through State Snapshot Transfer (SST)
  • Multi-threaded apply, greater write capacity than regular replication
  • Can do geographical disaster recovery (Geo DR)
  • More resilient to unresponsive nodes (swapping)
  • Can resolve split-brain situations by itself
  • Still under development, some rough edges
  • Large transactions like multi-statement transactions or large write operations cause issues and are usually not a good fit
  • For quorum reasons, 3 nodes are needed but one can be a lightweight arbitrator
  • SST can be heavy over a Wan
  • Commit are affected by the network latency, this impacts especially Geo DR
  • To achieve HA, a load balancer, like haproxy, is needed
  • Failover time is determined by the load balancer check frequency
  • Performance is affected by the weakest/busiest node
  • Foreign Keys are potential issues
  • MyISAM should be avoided
  • Can be mixed with regular async replication as master or slave but, slaves are not easy to reconfigure after a SST on their master
  • Require careful setup of the host, swapping can lead to node expulsion from the cluster
  • No manual failover mode
  • Debugging some Galera protocol issues isn’t trivial

Percona replication manager (PRM)

Percona replication manager (PRM) uses the Linux HA Pacemaker resource manager to manage MySQL and replication and provide high-availability. Information about PRM can be found here, the official page on the Percona web site is in the making.
POSITIVE POINTSNEGATIVE POINTS
  • Nothing specific regarding the workload
  • Unlimited number of slaves
  • Slaves can have different roles
  • Typically VIP based access, typically 1 writer VIP and many reader VIPs
  • Also works without VIP (see the fake_mysql_novip agent)
  • Detects if slave lags too much and remove reader VIPs
  • All nodes are monitored
  • The best slaves is picked for master after failover
  • Geographical Disaster recovery possilbe with the lightweight booth protocol
  • Can be operated in manual failover mode
  • Graceful failover is quick, under 2s in normal conditions
  • Ungraceful failover under 30s
  • Distributed operation with Pacemaker, no single point of failure
  • Builtin pacemaker logic, stonith, etc. Very rich and flexible.
  • Still under development, some rough edges
  • Transaction maybe lost is master crashes (async replication)
  • For quorum reasons, 3 nodes are needed but one can be a lightweight arbitrator
  • Only one node is writable
  • Read after write may not be consistent (replication lag)
  • Only scales reads
  • Careful setup for the host, swapping can lead to node expulsion from the cluster
  • Data inconsistency can happen if the master crashes (fix coming)
  • Pacemaker is complex, logs are difficult to read and understand

MySQL master HA (MHA)

Like with PRM above, MySQL master HA (MHA), provides high-availability through replication. The approach is different, instead of relying on an HA framework like Pacemaker, it uses Perl scripts. Information about MHA can be found here.
POSITIVE POINTSNEGATIVE POINTS
  • Mature
  • Nothing specific regarding the workload
  • No latency effects on writes
  • Can have many slaves and slaves can have different roles
  • Very good binlog/relaylog handling
  • Work pretty hard to minimise data loss
  • Can be operated in manual failover mode
  • Graceful failover is quick, under 5s in normal conditions
  • If the master crashes, slaves will be consistent
  • The logic is fairly easy to understand
  • Transaction maybe lost is master crashes (async replication)
  • Only one node is writable
  • Read after write may not be consistent (replication lag)
  • Only scales reads
  • Monitoring and logic are centralized, single-point of failure, a network partition can cause a split-brain
  • Custom fencing devices, custom VIP scripts, no reuse of other projects tools
  • Most of the deployments are using manual failover (at least at Percona)
  • Requires priviledged ssh access to read relay-logs, can be a security concern
  • No monitoring of the slave to invalidate it if it lags too much or if replication is broken, need to be done by external tool like HAProxy
  • Careful setup for the host, swapping can lead to node expulsion from the cluster

NDB Cluster

NDB cluster is the most high-end form of high-availability configuration for MySQL. It is a complete shared nothing architecture where the storage engine is distributed over multiple servers (data nodes). Probably the best starting point with NDB is the official document, here.
POSITIVE POINTSNEGATIVE POINTS
  • Mature
  • Synchronous replication
  • Very good at small transactions
  • Very good at high concurrency (many client threads)
  • Huge transaction capacity, more than 1M trx/s are not uncommon
  • Failover can be ~1s
  • No single point of failure
  • Geographical disaster recovery capacity built-in
  • Strong at async replication, applying by batches gives multithreaded apply at the data node level
  • Can scale reads and writes, the framework implements sharding by hashes
  • Not a drop-in replacement for Innodb, you need to tune the schema and the queries
  • Not a general purpose database, some loads like reporting are just bad
  • Only the Read-commited isolation level is available
  • Hardware heavy, need 4 servers mininum for full HA
  • Memory (RAM) hungry, even with disk-based tables
  • Complex to operate, lots of parameters to adjust
  • Need a load balancer for failover
  • Very new foreign key support, field reports scarce on it

Shared storage/DRBD

Achieving high-availability use a shared storage medium is an old and well known method. It is used by nearly all the major databases. The share storage can be a DAS connected to two servers, a LUN on SAN accessible from 2 servers or a DRBD partition replicated synchronously over the network. DRBD is by bar the most common shared storage device used in the MySQL world.
POSITIVE POINTSNEGATIVE POINTS
  • Mature
  • Synchronous replication (DRBD)
  • Automatic failover is easy to implement
  • VIP based access
  • Write capacity is impacted by network latency for DRBD
  • SANs are expensive
  • Only for InnoDB
  • Standby node, a big server doing nothing
  • Need a warmup period after failover to be fully operational
  • Disk corruption can spread

Tuesday 17 March 2015

10 JQGrid Tips learned from my experience

For the past few days I have been working with JQGrid, one of the most powerful JQuery plug-in.  To give an outline, JQGrid helps us to create a grid with ease. The power of JQGrid comes from the fact that almost all the needs for a grid are addressed using simple tunable options. JQGrid can be downloaded fromhttp://www.trirand.com/blog/?page_id=6.  JQGrid has a good documentation available at http://www.secondpersonplural.ca/jqgriddocs/index.htm. Also most of the JQGrid options are demonstrated well in http://www.trirand.com/jqgrid35/jqgrid.html. In spite of these well written documents, there are times I spent digging JQGrid source code for some customization needs. I also strongly feel that most of my co-workers might need the same at some point of time, so thought of blogging (documenting) for further reference.

All the code used in this article can be downloaded from here.
Our very first table looks as belowdraft-table
TIP 1: Changing JQGrid width and height
As depicted in the picture, the table doesn’t show up with paging information. Due to the smaller width, the 2nd column header and value are also not fully visible.  To solve this, we need to set the width of the table, thereby columns. This can be achieved as shown below
  1. During grid creation, use the property width with value in pixel as
1
width:700
OR
2. After grid creation, use setGridWidth(width, shrink) API as
1
$([tableid]).setGridWidth(700,true);
Similarly, height of the table can also be set using height :<value is pixel> and setGridHeight(height) API.  The default height is 150px. I am still searching for a way to set width/height of JQGrid in percentage (%). Comment, if you already know it.
TIP 2: Grid without unnecessary scroll area
If you are sure that the grid is not going to take more than N values and those N values can be fit in a single page, then there is no need for a scroll bar. In this case the space allotted for scrollbar is immaterial. I would rather prefer to remove this space. This can be achieved by setting the value of scrollOffset to zero as follows,
1
scrollOffset:0
By default the value of scrolloffset will be 18px TIP 3:  Error due to missing locale file
You have included all the needed JS files like, JQuery, jqgrid (single file as minimized js/jquery.jqGrid.min.js) but still hitting with an error “$.jgrid is undefined” and grid is not rendered. This might be due to missing locale file. To solve this include appropriate locale, as
1
<script src="jqgridsrc/src/i18n/grid.locale-en.js" type="text/javascript" charset="utf-8"></script>
TIP 4: Callback function
We came across a requirement where we need to associate a hover event only to the first cell of a grid. To do the same, we need to get the first cell (td) and associate hover function on it and then implementing the functionality needed. Let’s assume that these functionalities (writing selector to get the first cell and writing .hover(…) function) is available in a function named doWhenHover(). The point here is when to call this function? THINK !! THINK !! THINK !!
If you told that it should be after the function that creates the table, in our example createMyTable(), then you are wrong.  Since the grid is not guaranteed to be completely formed at the end of createMyTable() function. So, JQGrid provides a callback function, loadComplete to which a function can be associated during grid creation like,
1
2
3
4
5
6
7
8
9
10
11
loadComplete: doWhenGridFullyLoaded
….
//grid creation completed
function doWhenGridFullyLoaded(){
//do the operation here
}
TIP 5: Using pagers only for buttons (or icons)
JQGrid documentation provides a tip at http://www.secondpersonplural.ca/jqgriddocs/index.htm under jqGrid > User Modules > Tips, Tricks and Hacks using which pager can be removed.  Here is yet another way to do the same. If you want to have pager only to add buttons and don’t like to display page numbers and other stuffs then we could remove those items as shown below,
1
2
3
4
5
6
7
$(‘#
<pagername>_center’).remove();
$(‘#
<pagername>_right’).remove();
Example: $('#mysimpletablepager_right').remove();
TIP 6: Hiding unwanted buttons in navigation bar
When navGrid(…) API is called to add a icon, say refresh  as
1
2
$(‘#
<tableid>’).navGrid('#mysimpletablepager',{refresh:true});
the following icons will be added in addition to the refresh icon
edit, add, del, refresh, search
To hide any of these icons associate those icons to a value of false. Say for example to hide edit, add icons and show other icons we can use the following fragment of code
1
2
$(‘#
<tableid>’).navGrid('#mysimpletablepager',{edit:false,add:false});
Also, by default view icon will be hidden, this can be made visible by associating true to it as view:true
Tip 7: Hiding the inline search by default
In addition to the popup search in JQGrid there is also an additional option of having inline search. In this case, a search textbox will be added to top of each column. Refer JQGrid documentation for the procedure to add inline search. But, when we add inline search functionality the search textbox will be shown by default and it occupies the first row below the header.  Though it can be argued that, showing search textbox by default makes the functionality explicit to user, I like it to be the other way i.e. when the grid is loaded the search textbox has to be hidden. User can make explicit request to show the search textbox. To do the same call mygrid[0].toggleToolbar() (the same code that is called when inline search icon is clicked) at the end of grid creation code as follows,
1
2
3
4
5
6
7
8
9
10
11
12
13
14
var myGrid = $(‘#mysimpletable’).jqGrid({
     url:<value>,
    ….//.other options
   })
.navButtonAdd("<
<pagerid>>",{caption:"",title:"Toggle Search Toolbar", buttonicon :'ui-icon-search', onClickButton:function(){ mygrid[0].toggleToolbar() } }); //icon to add the search bar with search textbox
mygrid.filterToolbar(); //adds the search toolbar
mygrid[0].toggleToolbar();//hide the toolbar by default
TIP 8: Having your own icons for navigation bar buttons
We are not always fulfilled with the available JQGrid/JQuery-UI images. So we might need to put our own image/icon for some of the buttons. This could be achieved by writing our own CSS and associating it with the navigator grid button by specifying it as a value to buttonicon as shown below
1
2
3
4
5
6
7
8
9
.ui-icon.myicon {
    width: 16px;
    height: 16px;
    background-image:url(edit.png);
}
1
mygrid.navButtonAdd("#mysimpletablepager",{caption:"",title:"Toggle Search Toolbar", buttonicon:'myicon', onClickButton:function(){ mygrid[0].toggleToolbar() } });
TIP 9: Removing title bar close button
I don’t find a real use of the tile bar close button. It would be much useful if the adjacent components occupy the space whenever user shrinks the grid, something like IGoogle portlets. Though JQGrid does it, it largely depends on the enclosing component. So I prefer to remove of button thereby hiding the functionality from user. As other removal this can also be easily achieved using remove() API as shown below
1
$('.ui-jqgrid-titlebar-close','#gview_mysimpletable').remove();
Note: Title bar won’t appear if the caption is an empty string.
TIP 10: Adding toolbar
JQGrid also provides an option to add toolbar to the table. This can be achieved using toolbar option, toolbar:[true,”both”] during grid creation. First parameter specifies whether the grid needs to be added or not. The second parameter is to tell about the location of the toolbar. Second parameter can take three values namely, top, bottom and both.
Stay tuned to know about the procedure for adding buttons to toolbar.
Version of
JQGrid – 3.5 beta
JQuery – 1.3.2
JQuery UI Theme – Dark Hive
Feel free to share your experience with JQGrid and also if you have cleaner way to solve some of the items discussed above, as comments.

Angular Tutorial (Update to Angular 7)

As Angular 7 has just been released a few days ago. This tutorial is updated to show you how to create an Angular 7 project and the new fe...