Merle

The Merle locus (PMEL gene or SILV gene) is complex, both in the mechanism in the cell and the way the mutation works. The PMEL gene codes for a glycoprotein that is part of the cell membrane. One of the products that is formed when the glycoprotein is broken down is a building block for the structural matrix of the melanosomes (the cell organel where pigment is formed and stored). This matrix helps both with maintaining the shape of the cell as with the production of the pigment eumelanin. The mutation that results in the merle pattern inserts a block of DNA in this gene. This DNA has no functional meaning but it contains an area which is similar to the area in the original gene that marks the end of the gene. The way this is translated to mRNA and later on to a protein causes that both a short (functional) version and a long (disfunctional) version of the PMEL protein is generated. If the inserted DNA is short chances are pretty high that the original PMEL protein is formed and the longer the inserted DNA is the higher the chances are that the modified PMEL protein is generated. More broken protein means that cell structure and distribution of pigment in the cell are more affected. The pigment looks more diluted.

The cells from which pigment cells (melanocytes) are formed (melanoblasts) travel in the embryo from the neural crest (an area near the future spine) to the various areas in the skin. Each of those melanoblasts determines the coat colour for an area of the skin.

The merle mutation is a so called SINE — Short Interspersed Nuclear Elements. A SINE is a kind of parasitic piece of DNA that was inserted at some point in the past. Dog DNA contains a lot of SINEs; about 8% of the DNA. A SINE consists of a head, body and tail. The tail is a repeating section of base pairs. A SINE can generally be between 50 and 500 base pairs long. General DNA tests for the merle mutation only detect the body.

When dogs were DNA tested as merle but didn't show any merle patterns further research found that the length of the SINE correlated with certain phenotypes. Short insertions resulted in dogs that didn't show merle. When the merle mutation is not present the DNA test for the length of the insertion still measures something, but the number is always below 200 base pairs.
Note: in the publication by Murphy et al the length of the repetitive part of the tail is used. These numbers are 175 bp lower than the lenght of the entire insertion which is used by most researchers and labs.

Because certain ranges of lengths correspond with a number of distinct phenotypes a list of different "alleles" has been made. Below is the list made by Langevin et al. Not all publications agree with this but it's currently supported by a large number of DNA tests and photos in various breeds in the research by Langevin:

m : < 200 bp : non-merle
M^c : 200-230 bp : Cryptic merle
M^c+ : 231-246 bp : Cryptic+ merle
M^a : 247-254 bp : Atypical merle
M^a+ : 255-264 bp : Atypical+ merle
M : 265-268 bp : Classic merle
M^h : 269-280 bp : Harlequin merle

Mutations with lengths up til 246 base pairs (the alleles m, M^c and M^c+) do not result in any coat colour changes, except for M^c+/M^c+ which can show some minor diluted colours. M^h (Harlequin merle) is not the same as the Harlequin gene (H)! It results in a wide range of patterns. Combinations of M^h with other alleles cannot be distinguished only by the patterns they produce. The upper limit of 280 bp is not set in stone, some dogs are known to have even longer insertions.

From gene to pattern

The cells that form pigment cells (melanocytes) originate from an area in the embryo called the neural crest. This is an area in the early development of the embryo near the structure that later forms the spine. The cells in the neural crest migrate to various areas in the body. During the migration of melanoblasts (the precursor cells) the replication of areas with a repeating single base often goes wrong. In most cases the repeating sequence becomes shorter and sometimes longer. Each melanoblast will determine the colour of an area of the body. The earlier in the migration such a replication error occurs the larger the area with that colour becomes. The end result is a mix of larger and smaller areas with more or less diluted colours.

Double merle

Before we knew about the different tail lengths of the insertion homozygous merle dogs were thought to often have health problems. The alleles with short lengths however do not cause any issues. The combinations which can be problematic are: M^c+/M^a+, M^a+/M^a+, M^c+/M, M^a/M, M^a+/M, M/M, m/M^h, M^c/M^h, M^c+/M^h, M^a/M^h, M/M^h and M^h/M^h. They produce so much white in the coats that if that white is near the eyes or (inner) ears sight and/or hearing might be impaired.

If you look at the list of problematic combinations it can be summarized as: all combinations with harlequin merle (even non-merle!), classic merle with Cryptic+ or longer, Atypical+ with Cryptic+ or longer.

In a lot of countries breeding two merles is not allowed. With the current knowledge however we know that we can combine certain types of (genetic) merles without risking health issues. A complicating factor is that DNA tests are often done with cells from the inside of the mouth or with blood. Just like there are changes in the length of the insertion with replication during the migration of melanoblasts, mutations can also happen with other replications. Part of the eggs and sperm cells (the 'germ cells') may also have insertions with a different length than most other cells. Because we usually don't test germ cells (sperm could be tested but testing egg cells would be problematic) the results of DNA tests for the length(s) of the insertion may not always represent what is present in the germ cells. Although shortening is much more likely than elongation, the latter is still a possibility.

Mozaicism

Sometimes the length that is reported by a DNA test does not match with the observed pattern in an area of the body. In such a case parts of the body may have an allele with a different length than other parts. This effect is called mozaicism.

If you collect hairs from those different areas you can send them to a lab to determine the length of the tail section. In cases with mozaicism the lab result will show multiple lengths (e.g. m/M²⁵⁷/M²³⁶).

How to test

Most labs offer basic DNA tests for the body part of the insertion. Some do offer whole panels with tests for all colour genes and diseases. This is a good starting point. If the dog comes back as merle it becomes necessary to further test for the length of the mutation. Only a few labs offer this test, basically because the equipment is extremely expensive. In rare cases (see Mozaicism, above) it's necessary to test hairs from several areas of the coat.

Possible alleles:
See above.

Little, Clarence C., The Inheritance of Coat Color in Dogs, Ithaca, New York, Comstock Pub. Associates, 1957.

Brancalion, L., Haase, B. and Wade, C.M. (2022), Canine coat pigmentation genetics: a review. Anim Genet, 53: 3-34. https://doi.org/10.1111/age.13154

Candille SI, Kaelin CB, Cattanach BM, et al. A -defensin mutation causes black coat color in domestic dogs. Science. 2007;318(5855):1418-23. https://www.ncbi.nlm.nih.gov/pmc/articles/PMC2906624/

Murphy, Sarah C., Evans, Jacquelyn M., Tsai, Kate L. and Clark, Leigh Anne, Length variations within the Merle retrotransposon of canine PMEL: correlating genotype with phenotype, Mobile DNA20189:26, https://doi.org/10.1186/s13100-018-0131-6

Varga, László; Lénárt, Xénia; Zenke, Petra; Orbán, László; Hudák, Péter; Ninausz, Nóra; Pelles, Zsófia and Szőke, Antal; Being Merle: The Molecular Genetic Background of the Canine Merle Mutation, Genes 2020, 11(6), 660; https://doi.org/10.3390/genes11060660

Langevin, Mary; Synkova, Helena; Jancuskova,Tereza and Pekova, Sona, Merle phenotypes in dogs – SILV SINE insertions from Mc to Mh, September 20, 2018, https://doi.org/10.1371/journal.pone.0198536