Sunday, September 19, 2010

Tetra Beta OS 1.2.3b

Tetra Main OS 1.2.3:  Tetra_Main_1.2.3.syx
Tetra Voice OS 1.2.0: Tetra_Voice_1.2.0.syx

Some cool new stuff in this release, details below:
  1. Sequencer Slew added - Slew has been added as a Sequencer destination in Sequences 2 and 4 controlling the slew in Sequences 1 and 3 respectively
  2. Additional Arpeggiator Modes added: Now has Up, Down, Up/Down, Assign (hold the keys down), and Random for 1, 2 and 3 Octaves
  3. Individual Voice Volume control
  4. Multi Mode can now be controlled in full on the Tetra (details below)
  5. New clock mode: V1 Master
    • Voice 1 Tempo is used as the master, similar to MIDI IN
    • In Combo mode with V1 Master on, when you switch the program assigned to Voice 1 several parameters (Unison, A/B mode, Split, Arp and Seq settings and Play Button mode) are kept the same to keep the feel of the combo the same as you select new sounds
  6. New Global: Mode Lock = {Off, Program, Combo}
    • Locks the Interface to either Program or Combo mode so you don't mistakenly switch to the other mode using the program/combo switch.
    • When Mode Lock = Combo, the Tetra will start up in Combo mode
And some bug fixes:
  1. Mopho Keyboard programs now handled correctly
  2. USB Code improved, should help stability and jitter with newer Operating Systems
  3. Arpeggiator can now be turned On and Off via NRPN
  4. LFO Sync to MIDI clock fixed
  5. Inc/Dec buttons keep pitch offset
  6. Pots act correctly in Passthru and Relative modes when the value they are controlling is changed by another source (eg., MIDI or an AP Encoder)
  7. MIDI Clock now being sent at all times in Combo mode using the Voice 1 Clock
  8. In Combo mode, play button behavior when one or more voices was in Normal and the rest in Toggle mode fixed

 UI Details and Changes per mode:

Program Mode (Multi On):
  1. Current Voice displayed on the screen as M# in the upper right, similar to Combo mode
  2. Edit B now selects the next voice and cycles between voices 1 to 4
  3. All controls on the Tetra are locked to the active voice, including the play button
  4. The Global 'Write Basic Patch' writes the basic patch only to the active voice
  5. The Volume Pot controls the Preset Volume on the active voice. The Global Pot Mode setting controls this behavior, as per the Cutoff/Resonance pots
Combo Mode (Multi On and Off)
  1. The Tetra UI acts the same now whether Multi is on or off
  2. You can now change the program on each voice to other combo parts, which is the default behavior. When you are editing a single part of a combo on the name screen, the select encoder will change through combos 1-128, parts 1-4 incrementally. 
    1. Changing a program will not stop the other voices from playing, but will stop the active voice
    2. You can change between the combo and program banks by holding down the inc/dec buttons
  3. Select Encoder and Inc/Dec buttons control whatever is on the screen. If you just changed the Pitch, Attack or Dec/Rel encoders and multiple parameters are displayed, using the Select encoder or Inc/Dec buttons will change all parameters
  4. Name editing has been added for individual parts in the combo
  5. While editing all voices, the volume pot controls the Master volume (MIDI CC#7). While editing a single voice, the volume pot controls only the preset volume of the active voice. The Global Pot Mode setting controls this behavior, as per the Cutoff/Resonance pots.
  6. Voices are no longer turned off when switching to and from global, or when switching between voices
  7. While in Single Voice Edit, the play button controls only the active voice. In V1-4 Edit in Unison, if any voices are playing, the play button turns them all off. If no voices are playing, it starts them all normally.
Combo Mode (Multi On) MIDI Behavior as follows:
  1. Main Combo Editing V1-4
    1. Global Send set to NRPN
      1. Assigned Parameter Encoders send out NRPN on a single MIDI channel (AP 1 sends on the MIDI channel selected in the global settings, AP 2 sends on MIDI channel + 1, etc)
      2. Volume pot sends out CC#7 changes on all 4 channels
      3. All other controls send out a single NRPN on the main MIDI channel, per the Quad section in the manual
    2. Global Send set to CC
      1. Assigned Parameter Encoders send a CC out on a single MIDI channel
      2. Volume pot sends out CC#7 changes on all 4 channel
      3. Other controls send out 4 CC messages for each parameter changed
  2. Single Voice Editing
    1. Global Send set to NRPN
      1. Volume Pot sends out 1 NRPN for Preset Volume
      2. All other controls send out 1 NRPN as normal, on the MIDI channel of the active voice
    2. Global Send set to CC
      1. Volume Pot doesn't send anything
      2. Other controls send 1 CC if the parameter they are controlling has an assigned CC # on the MIDI channel of the active voice

Friday, June 4, 2010

Prophet '08 Main OS 2.2.7 Final

This is the final candidate for the Prophet '08 Main 2.2.7 and Voice 1.5 OS update

Change Log:
  1. Additional Arpeggiator modes added - additional octaves and random modes
  2. Sequencer Slew added -  You can now use a sequence to control the slew rate of the previous sequencer. Eg., if you set Sequencer 1 up then you can set the destination of sequencer 2 to the very last selection, 'Seq Slew' and vary the amount for the slew individually for each step of sequence 1
  3. Encoder response updated - This greatly improves the response of the encoders both for the new and old versions of the hardware. There is a new global value 'Enc Mode' where you can select between 'Detent' and 'No Detent' options for the responsiveness of the encoders. If you have one of the Pot Edition front panel sets with non-detented encoders for all encoders except Param 1/2, select the 'No Detent' option and it will correctly set the Param 1/2 encoders to detented response
  4. Several minor bug fixes
For those who are updating from a 1.X OS, this includes the other 2.0 Updates including 2 channel Multi Mode

Prophet '08 Keyboard OS 2.2.7: P8Key_Main_2.2.7.syx
Prophet '08 Rack OS 2.2.7: P8Rack_Main_2.2.7.syx
Prophet '08 Voice OS 1.5: P8_Voice_1.5.syx

Tuesday, May 4, 2010

Prophet '08 Beta Encoder update

Edit: Updated to Main OS version 2.2.5 and Voice OS 1.5

Now that the code is working I've run through this beta a few times over just to make sure it's functioning as is expected; so far so good! It has been improved across the board; slow, medium and fast response, acceleration and error correction. If you start using the beta I'd really appreciate any feedback on the responsiveness and how it feels in each of those modes. I won't be able to get it perfect for everybody and I'm pretty happy with how it feels here but I'd like a consensus that it's at least a pretty substantial improvement. If it's a bit off somewhere I'll definitely poke around a bit to see if I can tighten it up further.

I cannot guarantee that this will solve problems with encoders that have a mechanical problem. Please contact support@davesmithinstruments.com if you have encoders that continue to jump around or are unresponsive.

DO NOT USE THIS CODE IF YOU ABSOLUTELY NEED YOUR P08 WORKING!!! It's a Beta. If for some reason this causes your P08 to hang, it will be at least a few days before I can swap a board so you can get going again. So for now, don't update unless you can handle a few days without it.

I can't make an exhausted list of Sysex problems because, quite frankly, there are just too darned many issues I've seen. I'll probably put together a more full post on this and exactly how sysex is used (sort of like the NRPN posts below) with some examples and tips on editing and automating sysex changes for the Evolver line. For now, here's a few little Sysex and update related things to be aware of when doing your Prophet '08 OS update:

1. Make sure you can update the OS successfully with a Sysex file first. I don't have the time to troubleshoot all the possibilities and there are a lot of them. The biggest problem I've run into is running a class compliant USB MIDI device with Windows XP (it has OS related sysex issues with larger sysex files which I won't detail here) so don't even bother if you have one. You'll need drivers from the manufacturer that have the sysex bug worked out.
2. I have successfully loaded this OS using MIDI OX with Windows XP and Sysex Librarian on OSX. Those are the two Sysex utilities I'd recommend.
3. I do NOT recommend the Soundtower editors to do the sysex updates at this time. We've had some small errors with them in the past and I'm not confident the code has been worked out yet.
4. If on Mac OSX and it doesn't work the first time around with Sysex Librarian, try setting the transmit speed to 80%. That fixed most of the problems I was having with it
5. Certain interfaces have issues that are hard to pinpoint; here are a few we are aware of:
MOTU Ultralite Mk. 3 needs a driver update, the older ones have a very annoying intermittent bug with sysex transmissions
For some reason we can't get any Digidesign interfaces to correctly transmit the sysex files in Windows XP 



I created a google group for DSI Beta information which I may start using for comments and such, for now it's just an easy way to upload files and link them here. You can join there if you'd like to post direct comments about the OS but please keep other topics out for now.

Here's the Main OS 2.2.3 Beta: P8Key_Main_2.2.5.syx
And the Voice OS 1.5 Beta: P8Key_Voice_1.5.syx
And the thread in the Prophet 5/08 forum for discussion: Prophet '08 Beta Discussion

Sunday, March 28, 2010

NRPNs Part 3: Sending with Ableton and M4L

I realized that I didn't finish the NRPN saga with Ableton quite yet. We know how to filter out NRPN changes and use the reconstructed 14 bit values, but how do we send an NRPN change back to the device?

We'll use the same method as before, javascript objects, to split up an NRPN change into 4 CC packets to send back out to the device. Here's a snapshot of the Max device with the addition of the javascript object js nrpn_to_cc.js:


We'll go through the path really quickly...

MIDI comes in through midiin and into midiparse which filters different types of MIDI data to each of it's different outlet. Outlet 2 forwards all CC messages, so we connect that directly to js nrpn_filter.js to filter out NRPN messages. Any non-NRPN related CC messages are sent to outlet 0, which are displayed in the two number objects control change ctrl and value. When we receive 2-4 CC changes that represent an NRPN change we forward the index and value out of outlet 1, which are displayed in the two number objects NRPN change index and value.

After we've unpacked the CC data and taken a look at it, it's repacked and send to midiformat inlet 2, which takes a list of two values and turns it into a MIDI CC event. We do the same with the NRPN data but we have to translate it into the correct MIDI values first in js nrpn_to_cc.js, which sends out 4 CC messages for each NRPN change. midiformat then sends a raw MIDI stream to midiout and to a print object which spits out the raw MIDI to the main Max window for debugging.

The code in nrpn_to_cc.js is much simpler than the filter so we'll just start with pseudocode:

When we receive 2 values
    Split value 0 into two 7 bit values
    Send value 0 high 7 bits as a CC change to CC#99
    Send value 0 low 7 bits as a CC change to CC#98
    Split value 1 into two 7 bit values
    Send value 1 high 7 bits as a CC change to CC#38
    Send value 1 low 7 bits as a CC change to CC#6

Pretty simple! Since we already have the values we don't need to do a lot of voodoo to deal with timing. We could handle Running Status here and optimize the MIDI stream quite a bit but for now we'll just make sure we can reconstruct the NRPN packets to send out. The only real interesting part of this code for the budding code junkie is bitwise operators, so we'll go over those quick, give the full code and call it a day.

To get the high and low 7 bits we can either do some convoluted math to convert from base 2 to base 10 or just use bit operators to do it lickity split. Going back to our first example we'll use the value 572:

572 = 0x023C = 0b 00 0010 0011 1100

First we'll get the bottom 7 bits. We use the bit operator AND to grab certain bits from the number and ignore the rest. To demonstrate we'll do a couple bit operations...

First AND against all ones:
00 0010 0011 1100
AND11 1111 1111 1111
=00 0010 0011 1100

And second AND against all zeros:
00 0010 0011 1100
AND00 0000 0000 0000
=00 0000 0000 0000
Well now that we know if we AND a 1 to our original value, we get exactly what we put in. If we AND a 0 to the value, we always get 0. So what we want is to get the original value for the low 7 bits and ignore all the rest.This is called a bitmask.

So we'll do this for the lower 7 bits:
00 0010 0011 11000x023C
AND00 0000 0111 11110x007F
=00 0000 0011 11000x003C

 And for the higher 7 bits:
00 0010 0011 11000x023C
AND11 1111 1000 00000x3F80
=00 0010 0000 00000x0200
So now we have the lower 7 bits exactly where we want them, but the higher 7 bits aren't in the right place. We need to move them over 7 places to get it in the right position. Luckily there's an operator perfectly suited for this... >> 7 moves all the bits over by 7 places:

0b00 0010 0000 0000 >> 7 = 0b00 0000 0000 0001
0x0200 >> 7 = 0x0001

Now everything is ready to send out! Here's the code for the javascript object:

// MIDI CC constants
const CC_DATA_MSB = 6;
const CC_DATA_LSB = 38;
const CC_NRPN_LSB = 98;
const CC_NRPN_MSB = 99;

// inlets and outlets
inlets = 1;
outlets = 1;

// global variables and arrays
var nrpn_index;            // Save last known rpn index
var data;
var output;                // Up to 8 Bytes to send

// Maxobj variables for scripting
var nrpn_index_lsb;
var nrpn_index_msb;
var data_lsb;
var data_msb;

// methods start here

// list - expects an RPN 14 bit Index + 14 bit Value
function list(val)
{
    if(arguments.length==2)
    {
        nrpn_index = arguments[0];
        data = arguments[1];

        nrpn_index_lsb = nrpn_index & 0x007F;
        nrpn_index_msb = (nrpn_index & 0x3F8) >> 7 ;
        data_lsb = data & 0x007F;
        data_msb = (data & 0x3F8) >> 7;
       
        outlet(0, CC_NRPN_MSB, nrpn_index_msb);
        outlet(0, CC_NRPN_LSB, nrpn_index_lsb);
        outlet(0, CC_DATA_MSB, data_msb);
        outlet(0, CC_DATA_LSB, data_lsb);
    }
}

NRPNs Part 2: Filtering NRPNs in Ableton with M4L

Since the whole goal is getting Ableton controlling the Mopho and Tetra the next step is to figure out how the heck we can use NRPNs with Ableton. On the positive side, NRPNs are just CCs so Ableton doesn't filter them out like it does with Sysex. On the negative side, Ableton handles NRPNs changes as 4 separate CC changes which makes it darned near impossible to automate the parameters efficiently.

Enter Max for Live. M4L doesn't handle NRPNs natively but it does give us an easy way to intercept the stream of MIDI CCs we see and reconstruct it into 14 bit index + value pairs. There's a couple ways we can handle the NRPN translation in M4L: Create an object that watches for each CC change we are interested and constructs the data, use Javascript as a js object in Max to filter out the CCs, or write a module in C using the Max SDK. Since I like the flexibility of code with this type of problem and I'm not quite ready to explore the SDK (maybe in a few weeks) we're going to stick to javascript this time around.

Going from one of the MIDI tutorials I put together a Max patch that sends all the CC MIDI data from one input into a java script with 1 inlet and 2 outlets, intending to have outlet 0 forward all the CC data we didn't filter, and have each complete NRPN change sent to outlet 1.



Since we have some important CC values we're interested in  I saved them as constants in the beginning of nrpn_filter.js

const CC_DATA_MSB = 6;
const CC_DATA_LSB = 38;
const CC_NRPN_LSB = 98;
const CC_NRPN_MSB = 99;


Then you need to declare the number inlets and outlets for the Max object:

inlets = 1;
outlets = 2;


 At this point we have a totally useless object that looks like it did in the picture above. Time to get it to do something useful. Since the js nrpn_filter.js object will be getting data from the midiparse object I checked through the Max docs to find that midiparse, like many other objects, sends data out in the list format. In Max Javascript, if you create a function list(val) it will run every time you get a new input in the list format. That's pretty much what I want, to run some code to check the CC number and assemble the values every time we get a new CC change. So in nrpn_filter.js we'll create a skeleton for the list function:

function list(val)
{

}

Since I'm trying to be a better programmer lately I'm going to focus on taking small steps that create a working demo, first step will be just sending all the CC data out outlet 0 and doing a basic check to make sure we get what we expect. I also added a post() statement for debugging so I can see exactly what values go into the function printed to the main Max screen. Now our list functions looks like this:

function list(val)
{

    cc_index = arguments[0];
    cc_value = arguments[1];

    post(cc_index, cc_value);

    outlet(0, cc_index, cc_value);



}

After plugging in the Mopho Keyboard to USB and hooking this Max patch up to it, it's spitting out all the CCs I expect to see from each NRPN change. Great! To reconstruct the NRPN from CC messages we need to understand a little bit more about the way MIDI constructs it's data, in specific a shortcut MIDI uses to prevent redundant information called running status.

Running Status holds the last command that was sent over the MIDI bus so we don't have to send a new 0xB0 byte for each individual CC change on MIDI channel 1, we just send one 0xB0 and every data byte after that is interpreted as if it was a CC change on MIDI channel 1. So let's go back to the raw MIDI stream of the NRPN change we used in Part 1 (NRPN# 572 set to 163):

     B0 63 04 B0 62 3C B0 06 01 B0 38 23

If the device was using Running Status, the exact same command could look like this:

     B0 63 04 62 3C 06 01 38 23

Since MIDI is limited by the hardware specifications of the time it was created to send one 3 byte packet of data can take 1ms to transmit. Since we can only transmit one message at a time, if we're changing 10 CCs at once, it will be a minimum of 10ms from the first change to the last change, which could easily be noticed by the human ear. It's worse if during those changes you start inputting notes, which are even easier to hear than the generally more subtle CC changes. Luckily USB takes care of this since it's hundreds of times faster and you can bundle many MIDI messages in a single USB packet, and Max/Ableton will take care of that low level running status... so why am I bringing this up?

When we reconstruct the NRPN packet so far we've assumed that we're going to see 4 packets of data for each change. There a few cases that we may see:
  1. A device may only use the LSB values, not caring about the extra resolution of the MSB, so it will only send 2 packets per change
  2. A device may only use the LSB value of the index but will use the higher resolution of the Data, resulting in 3 packets
  3. A device may send 1-2 packets to indicate which NRPN it intends to change and then only send data changes from there on out, expecting that each new data packet will be interpreted as the same NRPN data change
  4. A device may send the packets in the order MSB then LSB, or the other way around
Even though I know exactly how the Mopho/Tetra are sending out MIDI we're going to assume that whatever device might be sending MIDI to this patch isn't as stable and we'll have to consider those edge cases. Since we're not entirely sure what order things will come in, we need to figure out what we are SURE is going to happen. The two things we can assume is that we will receive the NRPN index messages BEFORE the data and that the LSB of the data will change every time we get an NRPN change. So we'll use those rules to construct the algorithm. Here's the high level pseudocode for the reconstruction before we translate it into raw javascript:

If we receive an NRPN MSB or LSB index message
    Save the new NRPN MSB or LSB index
    Construct the new NRPN (even if we've only received one of the two messages)
If we receive an NRPN MSB data message
    If we have a valid NRPN index
        Save the new NRPN MSB data value
        If we have a valid NRPN LSB data value
            Construct the 14 bit data value  from the data MSB and data LSB
            Send a new NRPN message to outlet 1
    Else if we do NOT have a valid NRPN index
        It is not a valid NRPN data packet, forward the CC data to outlet 0
If we receive an NRPN LSB data message
    If we have a valid NRPN index
        Construct the 14 bit data value from the data MSB and data LSB
        Send a new NRPN message to outlet 1
    Else if we do NOT have a valid NRPN index
        It is not a valid NRPN data packet, forward the CC data to outlet 0
If we receive any other type of CC message
    Forward it to outlet 0

There's a catch here though. If we receive the data MSB and LSB out of order, we will end up sending 2 NRPN changes out. Since each one is 4 bytes, every time that happens we're adding almost 3ms of latency to a hardware MIDI bus. So in the interest of latency we'll assume that the data will arrive in MSB then LSB order.To determine when we've sent a valid NRPN message, since we are not sure if the MSB will change, we need to reset the data LSB to an invalid state every time we send a new NRPN change out.

Each time we send an NRPN message
    Reset the NRPN LSB data message to invalid

Here's the resulting nrpn_filter.js file:

// MIDI CC constants
const CC_DATA_MSB = 6;
const CC_DATA_LSB = 38;
const CC_NRPN_LSB = 98;
const CC_NRPN_MSB = 99;

// inlets and outlets
inlets = 1;
outlets = 2;

// global variables and arrays
var nrpn_index;

var value;
var nrpn_index_lsb;
var nrpn_index_msb = 0;
var value_lsb;
var value_msb = 0;
var cc_index;
var cc_value;

// methods start here

// list - expects a CC Index + Value as argument
// filters out NRPN and RPN values and assigns to variables
// Passes through all other CCs
function list(val)
{
    if(arguments.length==2)
    {
        cc_index = arguments[0];
        cc_value = arguments[1];

        switch(cc_index)
        {
        case CC_DATA_MSB:
            // If we have a valid NRPN index, then the data is valid
            if(nrpn_index)
            {
                value_msb = cc_value << 7;
                if(value_lsb)
                {
                    value = value_msb | value_lsb;
                    // We are now ready to send an index and value out!
                    send_nrpn(nrpn_index, value);
                }
            }
            // If we don't have an index it's invalid
            else
            {
                // So forward the value as a normal CC
                send_cc(cc_index, cc_value);
            }
            break;
        case CC_DATA_LSB:
            // If we have a valid NRPN index, then the data is valid
            if(nrpn_index)
            {
                value_lsb = cc_value;
                value = value_msb | value_lsb;
                // We are now ready to send an index and value out!
                send_nrpn(nrpn_index, value);
            }
            // If we don't have an index it's invalid
            else
            {
                // So forward the value as a normal CC
                send_cc(cc_index, cc_value);
            }
            break;
        // NRPN Index MSB - 7 high bits
        case CC_NRPN_MSB:
            // Save 7 high bits
            nrpn_index_msb = cc_value << 7;
            // Create the 14 bit NRPN index
            nrpn_index = nrpn_index_msb | nrpn_index_lsb;
            break;
       // NRPN Index LSB - 7 low bits
        case CC_NRPN_LSB:
            // Save 7 low bits
            nrpn_index_lsb = cc_value;
            // Create the 14 bit NRPN index
            nrpn_index = nrpn_index_msb | nrpn_index_lsb;
            break;
        default:
            send_cc(cc_index, cc_value);
            break;
        }
    }
}

// Send CC: Forward CC index and value to output 0
function send_cc(i, v)
{
    outlet(0, i, v);
}

// Send NRPN: Send NRPN index and value to output 1
function send_nrpn(i, v)
{
    outlet(1, i, v);
    value = null;        // reset the parsed value
    value_lsb = null;    // reset the LSB of the value
}

As a last word, I wouldn't consider myself a particularly good coder. I am aware that a lot of things here are redundant or inefficient but I tried to err on the side of readability. If you need a faster or more efficient handling (for a very large data stream for example) you'll probably want to rewrite this as a C module using the Max SDK.

If you spot any bugs or things that don't make sense, please let me know!

That's it for now, if I can make some progress on the Sysex side of things we'll dive into that next

NRPNs Part 1: The Basics

I started working on a Mopho/Tetra patch for Ableton Live now that I have Max for Live up and running. It's really a very cool level of integration, there's tons you can do with the two together! Of course since Ableton doesn't have fully featured MIDI handling you quickly run into some limitations...

For part 1 I'll give a little background for those who aren't intimately familiar with the MIDI specification:

Normally when you turn a knob or move a slider, you're changing a MIDI Control Change message (CC for short) which is the standard for controlling values on a synth. This is great for most things: since MIDI uses 7 bit numbers for data, you have up to 128 different parameters you can change. and each has 128 different values you can move it to (0 to 127). 128 is a good amount of parameters we can use, but the MIDI spec takes a handful of those and assigns them to predetermined or special functions so we lose almost 30 or so and we're down to 100 usable parameters in a standard setup.

100 parameters and each from 0 to 127... so what happens if we want to control more than 100 parameters or we have a parameter with a very large range that goes past 127? Lots of synths are coming out with more complicated controls and parameters and it's difficult to limit control to just 100 values. When we need more, we have a couple options:
  1. Remapping existing values (commonly pitchbend and poly key pressure) - a pitchbend message has a resolution of 14 bits. That means we have a range of 0 to 16383, plenty for a high resolution message. However we only have one pitchbend message per channel, so we're limited to a maximum of 16 high resolution parameters this way. This works pretty well if you have a small number of parameters you need increased resolution for. The best example of this is in any Mackie control surface: Mackie uses pitchbend messages for each volume slider to increase the resolution. Poly key pressure has a range of only 0-127 but we have a pitch assigned to each one as well, so it gives us a possible 0-127 new parameters with a range of 0-127. Some DAWs and controllers don't support Poly key pressure though as it's a fairly uncommon feature in modern keyboard controllers.
  2. Sysex - Sysex messages can have any number of bytes in them so we have tons of room to play around with things. We can have as many parameters and whatever resolution we desire! This is how it's done in the Evolver synths. The problem is since you can do anything with it each instrument normally has it's own standard for how it trasmits data. When you have your own standard you have to have some way to create the messages and interpret the messages. Most controllers and DAWs offer limited (if any) support for Sysex construction and deconstruction of values. You can use an interpreter like the powerful Bome's Midi Translator for Windows and Mac or a more simple free utility such as MIDIOX for Windows or MIDI Pipe for Mac OSX to do the translation. This is a lot of work to do, especially when you have to create a different mapping for every device you want to use, plus a lot of controllers and DAWs don't support Sysex at all making it impossible to set it up simply.
  3. NRPN - The MIDI protocol's answer to needing higher resolution values is Non-registered Parameter Numbers. An NRPN is essentially 4 CC number and value pairs that are combined to give much higher resolution. You have a full range of 0 to 16383 for 16384 different parameters we can control. Perfect! Well almost...
To find out what the problems are we need to understand what exactly an NRPN is and how to use them. So let's start by breaking down what an NRPN message looks like in MIDI. Here's an NRPN message in raw MIDI bytes for changing NRPN #572 to a value of 163 on MIDI channel 1:

     B0 63 04 B0 62 3C B0 06 01 B0 38 23

When we look at a MIDI stream like this, to determine if we're seeing a command (which MIDI refers to as 'status') or data we see if the first bit in the byte is a 1. This means 80-FF are all command bytes and the start of a new MIDI command. In our stream, the only bytes we have in that range are the bytes 'B0'. Looking at the MIDI spec, a command byte has two components to it, the first 4 bits indicate what command we are running and the last 4 bits indicate which MIDI channel we're on. So let's divide 'B0' up into parts:

    B = first 4 bits = Control Change (CC)
    0 = last 4 bits = MIDI Channel 1

That means we're seeing 4 Control Change messages for one NRPN change. Now let's figure out what CCs are being changed. First we'll divide the stream into the 4 individual CC Packets:

    B0 63 04
    B0 62 3C
    B0 06 01
    B0 26 23

Each Control Change message has 2 bytes of data following it, the first byte telling us which CC# we are changing and the second byte telling us the value that we are changing to. Since all the values are in hexadecimal form we'll also convert them to decimal to put them in a more readable form. The easiest way of doing this is in google, add a 0x to the beginning of each number to indicate that it's in hex form and enter in google '0x63 to decimal' which returns the decimal value of 99. So all our packets, expanded:

    Packet             Hex                     Decimal
    B0 63 04 - CC 0x63 = 0x04 -> Set CC# 99 to 4
    B0 62 3C - CC 0x62 = 0x3C -> Set CC# 98 to 60
    B0 06 01 - CC 0x06 = 0x01 -> Set CC# 6 to 1
    B0 26 23 - CC 0x26 = 0x23 -> Set CC# 38 to 35

So now we know exactly what it looks like when an NRPN is changed, but it's still pretty confusing... what do each of the 4 numbers represent? The first thing we need to figure out is exactly what those CC#s mean. Each of the 4 CC#s we see are assigned by the MIDI spec to special values for high precision changes. Going to the spec (A good list of MIDI messages and what they mean is here: http://www.midi.org/techspecs/midimessages.php) we find:

    CC# 99 (0x63) -  Non-registered Parameter Number Most Significant Byte (NRPN MSB)
    CC# 98 (0x62) -  Non-registered Parameter Number Least Significant Byte (NRPN LSB)
    CC# 6   (0x06) -  Data Entry Most Significant Byte (Data MSB)
    CC# 38 (0x26) -  Data Entry Least Significant Byte (Data LSB)

    Note: Sometimes documents will refer to MSB as 'Coarse' and LSB as 'Fine' for the Data Entry values

Looking at the names, it looks like we only really have 2 different parameters: NRPN and Data. Since each MIDI CC message has a maximum of 7 bits of data and an NRPN has 14 bits of data, we need to combine these messages to form the NRPN change. Let's go back to the data part of those messages now...

Remember how we found out which bytes were a command? We looked at the first bit to see if it was a 1. With a data packet, the first bit is always a 0. Since a MIDI packet is always 8 bits long, that means we lose 1 bit in every data packet to tell us whether or not it's a status or data byte, leaving us with 7 bits of useful data. So we can throw that first bit out. Now we have 7 bits + 7 bits = 14 bits, perfect! When we add them together the order of the bits will be MSB (Coarse changes) then LSB (Fine changes). To add them together we need to know what the 7 bits actually looks like for each value. We'll use the hex we found from the raw stream (0x##) and convert to binary (0b#### ####). The same trick with google can be used here, just enter '0x04 to binary':

    In Hex: CC# 0x63 to 0x04 ->  0x04 in binary is 0b0000 0100 so we take out the first bit and we end up with: 0x04 = 0b 000 0100

    NRPN MSB = 0x04 = 0b 000 0100
    NRPN LSB = 0x3C = 0b 011 1100
    Data MSB = 0x01 = 0b 000 0001
    Data LSB = 0x23 = 0b 010 0011

Now to glue them together we just write the 7 bits of the MSB then the 7 bits of the LSB to get the final 14 bit values:

    NRPN = 0b 00 0010 0011 1100 = 0x023C = 572
    Data = 0b 00 0000 1010 0011 = 0x00A3 = 163

At this point we know basically what's going on with NRPNs and what it looks like each time you send an update over MIDI. Next it's time to do something interesting now that we know what's going on.

Thursday, March 18, 2010

Prophet '08 Encoder Issues

Figure we'll start this blog off with the big issue, I'll see if I can explain the problem a little more clearly so people know what's going on and what the solutions are right now and what's coming up:

Physically the encoders work by several small metal "brushes" attached to the rotating shaft. Each one of these brushes slides over several spaced out metal plates as you turn the encoder. When a brush hits one of the plates it connects one of two signal pins coming out of the encoder to a high voltage, then when it slides off the plate, the voltage drops back to zero. The plates that the brushes slide over are spaced in such a way that when you turn the encoder it sends out pulses on the two pins that are phased either +90 degrees or -90 degrees depending on the direction you are spinning. This is known as a quadrature or incremental rotary encoder (http://en.wikipedia.org/wiki/Rotary_encoder#Incremental_rotary_encoder)

Anything interfering with the connection from the brush to the metal plate is going to cause the electrical connection to drop to zero very quickly. Instead of a nice clean pulse wave, we get a pulse with a lot of holes. Depending on the timing, this will look to the processor like a very quick direction change, a slightly longer direction change (about one full pulse length) or just totally invalid data. On the Prophet '08, as many of you well know, usually when this happens you seem to get stuck between two values and can't go up or down further until you slow down the turn. We'll call the interference between the brush and the metal plates 'noise' for now.

So we know that a noisy encoder will cause those jumpy values; the more noise, the more jumpy. To solve this problem let's start by having a look at what can cause that noise:
  1. Oxidation of the metal causes a jumpy electrical connection when the brush passes over due to patches of low connectivity on the metal surface.
  2. Turning the encoder faster causes the brushes to bounce up and down slightly as they hit the turn, if they bounce too high they lose connection with the plate and breaks the electrical connection.
  3. Pulling up on the encoder shaft can cause the brushes to hit less solidly as well because of the slight give in the housing.
  4. There is a lubricant/grease inside each encoder that helps the brushes connect. If this is removed or washed off it can have severe effects on the output. It also slowly wears off due to oxidation and just drying up as the encoder ages.
So what can we do to solve this? We want to do anything we can to improve the electrical connection of the spinning brushes. Going backwards from what causes the noise we have two main hardware solutions: bend or push the brushes down harder against the plates as they turn (but not too hard to cause more bouncing) or add connectivity in the form of a lubricant that will prevent the brushes from bouncing and improve the electrical conductivity when they do pass over the plates.

This is why we started sending out Deoxit tubes to anyone who had the encoder issues. We found that applying a couple drops of deoxit to all the encoders fixes close to 95% of the boards we've seen. It does a good job of removing the oxidation and keeps the brushes sliding over the surface smoothly. It's likely that a batch of encoders had the lubricant washed off and putting a little additional on will get it running smoothly again.

However there are the more extreme cases where deoxit does not fix the problem. Since we test the encoders before sending the instruments out it's likely they looked OK during testing but after a little oxidation in the real world and some mild usage and the problem reared it's ugly head. Maybe the metal of the brushes is a bit weaker in that batch, maybe the deoxit isn't strong enough to take all the oxidation off, we don't know. At this point we have only a couple options... replace all the encoders or look a bit deeper and figure out how to compensate in software.

I've been working on a software fix for the encoder issues, both preventing a low to mid grade of noise from causing the encoders to get stuck on values or change direction and a more general revamp of the encoder processing to improve the responsiveness. I'm a knob tweaker at heart and not being able to do a filter sweep with a flick of the fingers on a Prophet '08 breaks my heart. Sadly, It has turned out to be a much more complicated issue than hoped so it's taken a while to iron out. But do not lose hope! The Mopho Keyboard will have greatly improved responsiveness on the encoders which will be ported back as best I can. The response needs to be calibrated for the non-detented encoders and I'll run it against bad boards I've received to see if I can get them responding at least in a stable way.

I'll post an update here as soon as I have a beta Prophet '08 and Tetra OS ready with some encoder improvements to get a little feedback on the responsiveness.