Send feedback

Getting Started

The Serenade Protocol is a powerful way to write your own plugins and custom integrations using Serenade. In this guide, you'll see how you can create your own application to handle commands that are spoken into the Serenade app.

Complete sample code for a few different example applications using the Serenade Protocol can be found at https://github.com/serenadeai/protocol.

Concepts

The Serenade Protocol defines how any application can receive data from the Serenade app (like a list of voice commands to execute) and send data to the Serenade app (like the source code to modify). All communication happens via JSON over WebSockets, so any language with support for those (common!) technologies can be used to create a Serenade plugin. You can think of the protocol as a simple message-passing framework, whereby Serenade can pass messages to your plugin, and your plugin can pass messages back to Serenade.

Serenade works by detecting which application on your desktop has focus, and then sending data over a WebSocket to the plugin listening for that application. The Serenade app runs a WebSocket server on localhost:17373, and "registering" your plugin with the Serenade app is as simple as opening a new WebSocket connection to localhost:17373. Each Serenade plugin also defines a regular expression that specifies which proesss it corresponds to. For instance, our Atom plugin tells the Serenade app that it should receive messages when an application whose process name matches atom is focused.

All messages sent over the protocol will be JSON-encoded data with two top-level fields:

  • message: The type of message is being sent.
  • data: The payload of the message.

As you'll see, every WebSockets request sent to your plugin and every response to the Serenade app will follow this format.

From a plugin's perspective, the lifecycle of a typical voice command looks like this:

  1. The user speaks a voice command.
  2. The Serenade app asks the plugin registered for the currently-focused app for information about the source code being edited, like its text, cursor position, and filename.
  3. The plugin sends back all of the information it has about the text or source code being edited.
  4. Using the data from the editor, the Serenade app performs some source code manipulation, like adding a function or deleting a line.
  5. The Serenade app sends back the result to the plugin for the currently-focused app.
  6. The plugin displays the resulting source code, cursor position, etc. in the editor.

Connecting to Serenade

Once your plugin starts up, we need to create a new WebSocket connection to the Serenade app. Then, you need to send an active message to the Serenade app, so it knows we exist. The active message should specify the following data:

  • id: A unique ID for this instance of the plugin. In some cases, users can open multiple windows of the same application and create multiple plugin instances—this all depends on each applications plugin API. This ID will be used by Serenade to keep track of this instance.
  • app: The name of your plugin. This will be displayed in the bottom-left of the Serenade app, so users know they're using your plugin.
  • match: A case-insensitive regular expression that matches your application name. For instance, a plugin for Atom would supply a match of atom, so when a process containing atom is in the foreground.

Here's a snippet that opens a WebSocket connection and sends the active message. Because we'll probably run this application from a terminal, we'll specify a match of term, which will match processes like terminal and iterm2.

const WebSocket = require("ws");

let id = Math.random().toString();
let websocket = new WebSocket("ws://localhost:17373");
websocket.on("open", () => {
  websocket.send(JSON.stringify({
    message: "active",
    data: {
      id,
      app: "demo",
      match: "term",
    }
  }));
});

If the Serenade app isn't running yet, then trying to connect to localhost:17373 will naturally fail. So, you might want your plugin to automatically try to reconnect, so it connects as soon as the Serenade app is started. Or, you might want to display an error to the user, with a button to manually reconnect. Which approach you use is up to you, and may be limited by the API of the application you're developing a plugin for.

One last note—if the application you're writing a plugin for defines events for windows coming into focus, you should also send an active event when the foreground window changes. This will tell Serenade which instance of the plugin it should message, so it doesn't try to make changes to a window that's in the background.

Sending Heartbeats

In order to keep the connection between your plugin and the Serenade app alive, you should send regular heartbeat messages. If Serenade hasn't receive a heartbeat from your plugin in over 5 minutes, it will consider the plugin idle, and remove it.

To send heartbeats on a regular interval, simply send a heartbeat message with the id of your plugin instance:

setInterval(() => {
  websocket.send(JSON.stringify({
    message: "heartbeat",
    data: {
      id,
    }
  }));
});

Handling Commands

Now that your plugin can connect to Serenade, let's implement support for responding to messages sent by Serenade. Keep in mind that not every plugin can respond to every possible command. For instance, if you're writing a plugin for an app that doesn't support multiple tabs, then supporting the NEXT_TAB command doesn't really make sense. If that's the case, then your plugin can simply ignore the message.

Remember, complete sample code for a few different example applications using the Serenade Protocol can be found at https://github.com/serenadeai/protocol.

Receiving Messages

So far, your plugin has only sent messages to Serenade. Now, let's implement receiving messages from the Serenade app so your plugin can respond to voice commands. To do so, we'll just need to listen for data sent over the WebSocket and then call APIs from the plugin accordingly.

The below snippet registers an event handler that fires when the plugin receives data from Serenade, decodes the JSON message, determines what type of command is being run, and then runs some plugin-specific code.

websocket.on("message", (message) => {
  const data = JSON.parse(message).data;
  for (const command of data.response.execute.commandsList) {
    switch (command.type) {
      case "COMMAND_TYPE_SWITCH_TAB":
        // here's where your plugin will actually do the work!
        editor.setActiveTab(command.index);
        break;
    }
  }

  websocket.send(JSON.stringify({
    message: "completed",
    data: {}
  }));
});

As you can see, data.response.execute.commandsList contains a list of command objects that should be executed by your plugin. Keep in mind that Serenade enables you to chain voice commands together (like "next tab line 3"), and commandsList might have more than one command to execute. command.type specifies what type of command should be executed—a full list can be found in theCommands Reference. Some commands also pass additional data. In this case, command.index specifies which tab index the plugin should bring to the foreground.

Inside of the case for the SWITCH_TAB message, the code editor.setActiveTab(command.index) is just a placeholder—that's where you'll want to make the relevant API call that's specific to your plugin.

Finally, once your plugin has finished processing all commands, send a completed message back to the Serenade app to indicate that your plugin is done.

Editor State

Next, let's implement a more complex message, which also happens to be one of the most important messages that the Serenade app will send your plugin: the GET_EDITOR_STATE message. As its name suggests, your plugin will receive this message when Serenade needs an updated version of the text being edited. For instance, in a code editor, this message is used to read the current source code contents, the name of the file being edited, the cursor position, and so on.

Let's update the above snippet to handle this new command type:

websocket.on("message", (message) => {
  const data = JSON.parse(message).data;
  for (const command of data.response.execute.commandsList) {
    switch (command.type) {
      case "COMMAND_TYPE_SWITCH_TAB":
        // here's where your plugin will actually do the work!
        editor.setActiveTab(command.index);
        break;
      case "COMMAND_TYPE_GET_EDITOR_STATE":
        websocket.send(JSON.stringify({
          message: "callback",
          data: {
            callback: data.callback,
            data: {
              "message": "editorState",
              // here's where your plugin will actually do the work!
              data: editor.getEditorState()
            }
          }
        }));
        break;
    }
  }

  websocket.send(JSON.stringify({
    message: "completed",
    data: {}
  }));
});

While Serenade doesn't expect a response for a SWITCH_TAB command, it does expect a response for a GET_EDITOR_STATE command, and it will wait for the plugin to send one. To respond to Serenade, specify a message type of callback and pass back the callback ID that was given in the original message from Serenade (in this example, contained in data.callback). Here, the inner data field also contains the relevant data from the active text field.

Most commands don't expect any response, but all of the expected responses are documented in theCommands Reference.

You'll notice that when the Serenade app is active, it will regularly send GET_EDITOR_STATE messages to make sure that the app has the most up-to-date version of the language being used in the editor. As long as your plugin responds to these messages, Serenade will automatically use the correct language.

Complete Example

Let's put everything together into a single example. We've also added some extra error checking and factored out some common code to make things easier to read. This example can also be found at https://github.com/serenadeai/protocol.

const WebSocket = require("ws");

const id = Math.random().toString();
let websocket = null;

const connect = () => {
  if (websocket) {
    return;
  }

  websocket = new WebSocket("ws://localhost:17373");

  websocket.on("open", () => {
    send("active", {
      id,
      app: "demo",
      match: "term",
    });
  });

  websocket.on("close", () => {
    websocket = null;
  });

  websocket.on("message", (message) => {
    handle(message);
  });
};

const handle = (message) => {
  const data = JSON.parse(message).data;
  if (!data.response || !data.response.execute) {
    return;
  }

  for (const command of data.response.execute.commandsList) {
    switch (command.type) {
      case "COMMAND_TYPE_GET_EDITOR_STATE":
        send("callback", {
          callback: data.callback,
          data: {
            message: "editorState",
            // here's where your plugin will actually do the work!
            data: { source: "", cursor: 0, filename: "" },
          },
        });
        break;
      case "COMMAND_TYPE_SWITCH_TAB":
        // here's where your plugin will actually do the work!
        break;
    }
  }

  send("completed", {});
};

const send = (message, data) => {
  if (!websocket || websocket.readyState != 1) {
    return;
  }

  try {
    websocket.send(JSON.stringify({ message, data }));
  } catch (e) {
    websocket = null;
  }
};

const start = () => {
  connect();
  setInterval(() => {
    connect();
  }, 1000);

  setInterval(() => {
    send("heartbeat", {
      id,
    });
  }, 60 * 1000);
};

start();

Commands Reference

Below is a list of all of the commands that the Serenade app can send your plugin. There's no need to respond to every message type—if a particular message doesn't make sense for your plugin, then you can simply ignore it.

The most important command types are COMMAND_TYPE_GET_EDITOR_STATE and COMMAND_TYPE_DIFF, which are used for reading and writing text. At a minimum, your plugin should support these two commands.

COMMAND_TYPE_GET_EDITOR_STATE

Get the current contents of the editor.

  • limited If true, then only return the filename of the editor. This is used by the Serenade app for language detection.

COMMAND_TYPE_DIFF

Set the current contents of the editor.

  • source The new source code the editor should display.
  • cursor The new cursor position, expressed as an index into the source.

COMMAND_TYPE_CREATE_TAB

Create a new tab.

COMMAND_TYPE_CLOSE_TAB

Close the current tab.

COMMAND_TYPE_NEXT_TAB

Switch to the next tab.

COMMAND_TYPE_PREVIOUS_TAB

Switch to the previous tab.

COMMAND_TYPE_SWITCH_TAB

Switch to the tab at the given index.

  • index The index of the tab to switch to.

COMMAND_TYPE_OPEN_FILE_LIST

Get a list of files matching the given search string. Any filename with the search string as a substring should match.

  • path The path to search

COMMAND_TYPE_OPEN_FILE

Open the file at the given index in the file list returned by the previous COMMAND_TYPE_OPEN_FILE_LIST command.

  • index The index of the file to open

COMMAND_TYPE_UNDO

Undo the previous editor action.

COMMAND_TYPE_REDO

Redo the previous editor action.

COMMAND_TYPE_SAVE

Save the current file.

COMMAND_TYPE_SCROLL

Scroll the app.

  • direction Direction to scroll.

COMMAND_TYPE_STYLE

Invoke the editor's built-in styler on the current file.

COMMAND_TYPE_GO_TO_DEFINITION

Go to the definition of the symbol under the cursor.

COMMAND_TYPE_DEBUGGER_TOGGLE_BREAKPOINT

Toggle a breakpoint at the current line.

COMMAND_TYPE_DEBUGGER_START

Start the debugger.

COMMAND_TYPE_DEBUGGER_PAUSE

Pause the debugger.

COMMAND_TYPE_DEBUGGER_STOP

Stop the debugger.

COMMAND_TYPE_DEBUGGER_SHOW_HOVER

While debugging, show debug information for the current symbol.

COMMAND_TYPE_DEBUGGER_CONTINUE

While debugging, continue to the next breakpoint.

COMMAND_TYPE_DEBUGGER_STEP_INTO

While debugging, step into the current line.

COMMAND_TYPE_DEBUGGER_STEP_OUT

While debugging, step out of the current function.

COMMAND_TYPE_DEBUGGER_STEP_OVER

While debugging, step over the next line.

COMMAND_TYPE_DEBUGGER_INLINE_BREAKPOINT

Toggle an inline breakpoint.

COMMAND_TYPE_SELECT

Select a block of text.

  • cursor Index of the start of the selection block.
  • cursorEnd Index of the stop of the selection block.

COMMAND_TYPE_CLICK

For browsers, click an element on the current page.

  • text Text to click on the page.

COMMAND_TYPE_BACK

For browsers, go to the previous page.

COMMAND_TYPE_FORWARD

For browsers, go to the next page.

COMMAND_TYPE_RELOAD

For browsers, reload the current page.

COMMAND_TYPE_SHOW

For browsers, show links that can be clicked, inputs that can be focused, or code that can be copied.

  • text What type of elements to show, either links, inputs, or code.

Messages Reference

Below is a list of all of the messages that can be sent over the Serenade Protocol.

active

Sent by a plugin when a plugin first starts, a new window is opened, or multiple windows are open and the foreground window changes.

  • id A unique identifier for this plugin & window. Often, plugins just generate a random string.
  • app The human-readable name of the app this plugin is for. This name will be displayed at the bottom-left of the Serenade app.
  • match A regular expression describing what processes this plugin is active for. This field is used by the Serenade app to know which plugin it should forward messages to, based on the process name of the foreground application. For instance, our Atom plugin uses a match of atom, so it will match any process with atom in its name.

heartbeat

Sent by a plugin to keep the connection alive with the Serenade app.

  • id The ID of the current plugin.

response

Sent by the Serenade app when it needs the plugin to execute a sequence of commands.

  • response An object containing a list of commands to execute.callback A unique ID for the list of commands, which can be sent back to the Serenade app via a callback message as a means of responding to the Serenade app.

callback

Sent by a plugin to respond to a message from the Serenade app. Possible values are:

  • { message: "editorState", data: { source, cursor }}: In response to a COMMAND_TYPE_GET_EDITOR_STATE command, send a message of editorState and define the source and cursor in the data field.

completed

Sent by a plugin when it has finished executing commands sent by the Serenade app.