The Serenade Protocol defines how any application can receive data from the Serenade app (like a list of voice commands to execute) and send data to the Serenade app (like the source code to modify). All communication happens via JSON over WebSockets, so any language with support for those (common!) technologies can be used to create a Serenade plugin. You can think of the protocol as a simple message-passing framework, whereby Serenade can pass messages to your plugin, and your plugin can pass messages back to Serenade.

Serenade works by detecting which application on your desktop has focus, and then sending data over a WebSocket to the plugin listening for that application. The Serenade app runs a WebSocket server on localhost:17373, and "registering" your plugin with the Serenade app is as simple as opening a new WebSocket connection to localhost:17373. Each Serenade plugin also defines a regular expression that specifies which proesss it corresponds to. For instance, our Atom plugin tells the Serenade app that it should receive messages when an application whose process name matches atom is focused.

All messages sent over the protocol will be JSON-encoded data with two top-level fields:

• message: The type of message is being sent.
• data: The payload of the message.

As you'll see, every WebSockets request sent to your plugin and every response to the Serenade app will follow this format.

From a plugin's perspective, the lifecycle of a typical voice command looks like this:

1. The user speaks a voice command.
2. The Serenade app asks the plugin registered for the currently-focused app for information about the source code being edited, like its text, cursor position, and filename.
3. The plugin sends back all of the information it has about the text or source code being edited.
4. Using the data from the editor, the Serenade app performs some source code manipulation, like adding a function or deleting a line.
5. The Serenade app sends back the result to the plugin for the currently-focused app.
6. The plugin displays the resulting source code, cursor position, etc. in the editor.

Once your plugin starts up, we need to create a new WebSocket connection to the Serenade app. Then, you need to send an active message to the Serenade app, so it knows we exist. The active message should specify the following data:

• id: A unique ID for this instance of the plugin. In some cases, users can open multiple windows of the same application and create multiple plugin instances—this all depends on each applications plugin API. This ID will be used by Serenade to keep track of this instance.
• app: The name of your plugin. This will be displayed in the bottom-left of the Serenade app, so users know they're using your plugin.
• match: A case-insensitive regular expression that matches your application name. For instance, a plugin for Atom would supply a match of atom, so when a process containing atom is in the foreground.
• icon: The application icon to show in the main Serenade window when the application is active. Must be encoded as a data URL string and cannot be more than 20,000 characters long; ideally use a small (less than 48x48 pixels) version of the application icon. This field is only needed in the first active message or whenever the icon changes (e.g. to show custom status icons), and may be left out entirely if an icon isn't requred.

Here's a snippet that opens a WebSocket connection and sends the active message. Because we'll probably run this application from a terminal, we'll specify a match of term, which will match processes like terminal and iterm2.

const WebSocket = require("ws");

let id = Math.random().toString();
let websocket = new WebSocket("ws://localhost:17373");
websocket.on("open", () => {
  websocket.send(JSON.stringify({
    message: "active",
    data: {
      id,
      app: "demo",
      match: "term",
    }
  }));
});

If the Serenade app isn't running yet, then trying to connect to localhost:17373 will naturally fail. So, you might want your plugin to automatically try to reconnect, so it connects as soon as the Serenade app is started. Or, you might want to display an error to the user, with a button to manually reconnect. Which approach you use is up to you, and may be limited by the API of the application you're developing a plugin for.

One last note—if the application you're writing a plugin for defines events for windows coming into focus, you should also send an active event when the foreground window changes. This will tell Serenade which instance of the plugin it should message, so it doesn't try to make changes to a window that's in the background.

In order to keep the connection between your plugin and the Serenade app alive, you should send regular heartbeat messages. If Serenade hasn't receive a heartbeat from your plugin in over 5 minutes, it will consider the plugin idle, and remove it.

To send heartbeats on a regular interval, simply send a heartbeat message with the id of your plugin instance:

setInterval(() => {
  websocket.send(JSON.stringify({
    message: "heartbeat",
    data: {
      id,
    }
  }));
});

So far, your plugin has only sent messages to Serenade. Now, let's implement receiving messages from the Serenade app so your plugin can respond to voice commands. To do so, we'll just need to listen for data sent over the WebSocket and then call APIs from the plugin accordingly.

The below snippet registers an event handler that fires when the plugin receives data from Serenade, decodes the JSON message, determines what type of command is being run, and then runs some plugin-specific code.

websocket.on("message", (message) => {
  const data = JSON.parse(message).data;
  let result = {
    message: "completed"
  };

  for (const command of data.response.execute.commandsList) {
    switch (command.type) {
      case "COMMAND_TYPE_SWITCH_TAB":
        // here's where your plugin will actually do the work!
        editor.setActiveTab(command.index);
        break;
    }
  }

  websocket.send(JSON.stringify({
    message: "callback",
    data: {
      callback: data.callback,
      data: result
    }
  }));
});

As you can see, data.response.execute.commandsList contains a list of command objects that should be executed by your plugin. Keep in mind that Serenade enables you to chain voice commands together (like "next tab line 3"), and commandsList might have more than one command to execute. command.type specifies what type of command should be executed—a full list can be found in theCommands Reference. Some commands also pass additional data. In this case, command.index specifies which tab index the plugin should bring to the foreground.

Inside of the case for the SWITCH_TAB message, the code editor.setActiveTab(command.index) is just a placeholder—that's where you'll want to make the relevant API call that's specific to your plugin.

Serenade expects a reply for each message it sends to your plugin. Each request will contain a callback field that uniquely identifies that request. After you've handled a request, your plugin should send a callback message with the value of the request's callback field in the data object. Here, we specify "message": "completed" to indicate that the command ran successfully.

Next, let's implement a more complex message, which also happens to be one of the most important messages that the Serenade app will send your plugin: the GET_EDITOR_STATE message. As its name suggests, your plugin will receive this message when Serenade needs an updated version of the text being edited. For instance, in a code editor, this message is used to read the current source code contents, the name of the file being edited, the cursor position, and so on.

Let's update the above snippet to handle this new command type:

websocket.on("message", (message) => {
  const data = JSON.parse(message).data;
  let result = {
    message: "completed"
  };

  for (const command of data.response.execute.commandsList) {
    switch (command.type) {
      case "COMMAND_TYPE_SWITCH_TAB":
        // here's where your plugin will actually do the work!
        editor.setActiveTab(command.index);
        break;
      case "COMMAND_TYPE_GET_EDITOR_STATE":
        result = {
          message: "editorState",
          // here's where your plugin will actually do the work!
          data: editor.getEditorState()
        };
        break;
    }
  }

  websocket.send(JSON.stringify({
    message: "callback",
    data: {
      callback: data.callback,
      data: result
    }
  }));
});

While Serenade doesn't expect a response for a SWITCH_TAB command, it does expect a response for a GET_EDITOR_STATE command, and it will wait for the plugin to send one. To respond to Serenade, specify a message type of callback and pass back the callback ID that was given in the original message from Serenade (in this example, contained in data.callback). Here, the inner data field also contains the relevant data from the active text field.

Most commands don't expect any response, but all of the expected responses are documented in theCommands Reference

You'll notice that when the Serenade app is active, it will regularly send GET_EDITOR_STATE messages to make sure that the app has the most up-to-date version of the language being used in the editor. As long as your plugin responds to these messages, Serenade will automatically use the correct language.

Let's put everything together into a single example. We've also added some extra error checking and factored out some common code to make things easier to read. This example can also be found at https://github.com/serenadeai/protocol

const WebSocket = require("ws");

const id = Math.random().toString();
let websocket = null;

const connect = () => {
  if (websocket) {
    return;
  }

  websocket = new WebSocket("ws://localhost:17373");

  websocket.on("open", () => {
    send("active", {
      id,
      app: "demo",
      match: "term",
    });
  });

  websocket.on("close", () => {
    websocket = null;
  });

  websocket.on("message", (message) => {
    handle(message);
  });
};

const handle = (message) => {
  const data = JSON.parse(message).data;
  if (!data.response || !data.response.execute) {
    return;
  }

  let result = {
    message: "completed"
  };

  for (const command of data.response.execute.commandsList) {
    switch (command.type) {
      case "COMMAND_TYPE_GET_EDITOR_STATE":
        result = {
          message: "editorState",
          // here's where your plugin will actually do the work!
          data: editor.getEditorState()
        };
        break;
      case "COMMAND_TYPE_SWITCH_TAB":
        // here's where your plugin will actually do the work!
        break;
    }
  }

  send("callback", {
    callback: data.callback,
    data: result
  });
};

const send = (message, data) => {
  if (!websocket || websocket.readyState != 1) {
    return;
  }

  try {
    websocket.send(JSON.stringify({ message, data }));
  } catch (e) {
    websocket = null;
  }
};

const start = () => {
  connect();
  setInterval(() => {
    connect();
  }, 1000);

  setInterval(() => {
    send("heartbeat", {
      id,
    });
  }, 60 * 1000);
};

start();