Communicating with Shelly Devices via Bluetooth Low Energy (BLE) Using RPC

Overview

Bluetooth Low Energy (BLE) has become a cornerstone technology for wireless communication in modern smart devices, offering efficient and low-power connectivity. Shelly devices, integral to smart home ecosystems, leverage BLE to facilitate seamless communication and control. A key aspect of Shelly’s BLE communication is the implementation of Remote Procedure Call (RPC) protocols over BLE, enabling sophisticated interactions between devices and controllers. This article delves into the technical nuances of communicating with Shelly devices using BLE, focusing on the Generic Attribute Profile (GATT), handles, and the mechanisms for executing RPC-based read and write operations.

Prerequisites

Before delving into the technical aspects of communicating with Shelly devices via BLE and RPC, it’s essential to have a foundational understanding of the following concepts:

1. Bluetooth Low Energy (BLE): A wireless personal area network technology designed for low power consumption and cost. BLE is widely used in IoT devices for efficient communication.

2. Generic Attribute Profile (GATT): A BLE protocol that defines how data is structured and exchanged between devices. GATT organizes data into services and characteristics.

3. Remote Procedure Call (RPC): A protocol that allows a program to execute a procedure (subroutine) on a remote device as if it were a local call.

4. JSON: A lightweight data interchange format used for structuring RPC requests and responses.

Shelly’s BLE RPC Architecture

Communicating with Shelly devices via BLE and RPC involves a seamless integration of establishing a connection, interacting with specific GATT services and characteristics, constructing and sending RPC requests, and receiving and parsing responses. This unified workflow ensures efficient and reliable communication with Shelly devices, enabling diverse integrations across different systems and programming languages.

Key GATT Components in Shelly Devices

1. Shelly GATT Service UUID: Identifies the primary service responsible for RPC communication.

SHELLY_GATT_SERVICE_UUID = "5f6d4f53-5f52-5043-5f53-56435f49445f"

2. RPC Data Characteristic UUID: Handles the transmission of RPC request and response data.

RPC_CHAR_DATA_UUID = "5f6d4f53-5f52-5043-5f64-6174615f5f5f"

3. RPC TX Control Characteristic UUID: Manages the transmission control, particularly for sending data length information to the Shelly device.

RPC_CHAR_TX_CTL_UUID = "5f6d4f53-5f52-5043-5f74-785f63746c5f"

4. RPC RX Control Characteristic UUID: Handles reception control, primarily for receiving data length information from the Shelly device.

RPC_CHAR_RX_CTL_UUID = "5f6d4f53-5f52-5043-5f72-785f63746c5f"

Note: These UUIDs are specific to Shelly devices, refer to Shelly’s official documentation or use BLE scanning tools to identify the appropriate UUIDs.

Key Aspects of Shelly’s RPC over BLE

JSON-RPC 2.0 Protocol

Shelly’s Gen2+ devices utilize the JSON-RPC 2.0 protocol for monitoring and controlling functionalities. This protocol is symmetric, allowing both peers—the client and the device—to invoke methods and send notifications to each other. JSON-RPC 2.0 ensures structured and standardized communication, facilitating reliable interactions between the client application and the Shelly device.

Namespace Conventions

Shelly organizes its RPC methods into namespaces to categorize functionalities, enhancing clarity and maintainability. Below are some primary namespaces and example methods:

• Shelly Namespace: Handles system management, configuration, and status.

  • Shelly.FactoryReset

  • Shelly.ResetWiFiConfig

  • Shelly.ListTimezones

• Switch Namespace: Manages discrete power outputs, typically relays.

  • Switch.Set

  • Switch.GetConfig

Note: The above namespaces and methods are illustrative examples. Shelly’s RPC implementation includes numerous additional methods across various namespaces to support diverse device functionalities.

Frame Types

Communication between the user and the device consists of three types of frames:

1. Request Frame: Used to send commands to devices.

2. Response Frame: Used to receive replies from devices.

3. Notification Frame: Used to receive unsolicited updates from devices.

Frame Structures in Shelly’s RPC Protocol

1. Request Frame

The request frame is a JSON object used to invoke methods on Shelly devices. It contains the following attributes:

• jsonrpc (string): Specifies the version of JSON-RPC used, typically "2.0". This field may be omitted.

• id (number or string): Identifier for the request, used to match the response frame. Required.

• src (string): Name of the source of the request (e.g., "user_1"). Required.

• method (string): Name of the procedure to be called (e.g., "Shelly.GetDeviceInfo"). Required.

• params (object): Parameters that the method takes (e.g., {"id":1}). Optional.

2. Response Frame

The response frame is a JSON object returned by the Shelly device in response to a request frame. It contains the following attributes:

• id (number or string): Identifier of the communication. Required.

• src (string): Name of the source of the response (typically the Shelly device). Required.

• dst (string): Name of the destination (typically the requester). Required.

• result (object): The result of the invoked procedure if the request was successful. Mutually exclusive with error.

• error (object): Contains the description of the error that occurred if the request was unsuccessful. Mutually exclusive with result.

3. Notification Frame

The notification frame is a JSON object sent by the Shelly device to notify the client of certain events or status changes without expecting a response. It contains the following attributes:

• src (string): Name of the source of the notification (e.g., Shelly device). Required.

• dst (string): Name of the destination (e.g., "user_1"). Required.

• method (string): The method invoked (e.g., "NotifyStatus"). Required.

• params (object): The parameters of the notification containing relevant data. Required.

Step-by-Step Guide to Communicating with Shelly Devices

1. Establishing a BLE Connection

Objective: Connect to the Shelly device using its unique BLE address.

Process:

• Device Discovery: Begin by scanning for nearby BLE devices to identify the Shelly device. This can be achieved using BLE scanning tools or libraries available in your chosen programming environment. The Shelly device can be identified by its name or BLE address.

• Initiate Connection: Once identified, initiate a connection to the Shelly device using its BLE address. This connection forms the foundation for all subsequent interactions. Ensure that your system’s BLE hardware is functional and that the Shelly device is in a state ready to accept connections.

Considerations:

• Connection Timeouts: BLE connections can sometimes be unstable or take longer than expected. Implement timeout mechanisms to handle scenarios where the connection attempt exceeds a reasonable duration.

• Reconnection Logic: In cases of unexpected disconnections, having a reconnection strategy ensures that your application can recover gracefully without requiring manual intervention.

2. Interacting with GATT Services and Characteristics

Objective: Access and utilize the specific GATT services and characteristics required for RPC communication.

Process:

• Service Discovery: Once connected, perform service discovery to retrieve the list of available GATT services offered by the Shelly device. Locate the Shelly GATT service using its predefined UUID.

• Characteristic Discovery: Within the Shelly GATT service, identify the RPC Data, RPC TX Control, and RPC RX Control characteristics using their respective UUIDs. These characteristics are pivotal for sending RPC requests and receiving responses. Refer to Key GATT Components in Shelly Devices.

Considerations:

• UUID Accuracy: Ensure that the correct UUIDs are used to prevent miscommunication. Using incorrect UUIDs can lead to failures in reading from or writing to the desired characteristics.

• Characteristic Availability: Some Shelly devices might have variations in their GATT profiles based on the model or firmware version. Implement checks to confirm the presence of all required characteristics before proceeding.

3. Constructing RPC Requests

Objective: Create RPC requests in the correct format to communicate desired actions to the Shelly device.

Process:

• JSON Structure: RPC requests are structured as JSON objects containing specific fields:

• id: A unique identifier for the request, ensuring that responses can be correlated with their respective requests.

• src: Source identifier, such as “user_1”, indicating the origin of the request.

• method: The RPC method to invoke, for example, “Shelly.GetDeviceInfo”.

• params: Optional parameters required by the RPC method.

{
    "id": 123456789,
    "src": "user_1",
    "method": "Shelly.GetDeviceInfo",
    "params": {}
}

• Encoding: Convert the JSON object into a UTF-8 encoded byte array. This byte array is what will be transmitted over BLE to the Shelly device.

• Calculating Length: Determine the byte length of the encoded RPC request. This length information is crucial for informing the Shelly device about the size of the incoming data.

• Packing the Length: Convert the calculated length into a 4-byte big-endian integer. This packed length is written to the RPC TX Control characteristic to signal the incoming data size.

Considerations:

• Unique Request IDs: Each RPC request should have a unique id to accurately match responses with their corresponding requests.

• Parameter Validation: Ensure that the parameters provided to the RPC method are valid and adhere to the expected format. Invalid parameters can lead to malformed requests and errors from the Shelly device.

4. Sending RPC Requests

Objective: Transmit the constructed RPC request to the Shelly device via BLE.

Process:

• Write Length to TX Control Characteristic: Begin by writing the packed length of the RPC request to the RPC TX Control characteristic. This informs the Shelly device about the size of the incoming data, preparing it to receive the actual RPC request.

• Introduce a Short Delay: After writing the length, introduce a brief delay (e.g., 1 second) to allow the Shelly device to process the length information. This ensures synchronization between the client and the device before sending the actual data.

• Write RPC Request to Data Characteristic: Proceed to write the encoded RPC request bytes to the RPC Data characteristic. This action sends the actual command or query to the Shelly device, prompting it to perform the specified RPC method.

Considerations:

• Write Operations with Response: Utilize write operations that expect a response (acknowledgment) from the Shelly device. This ensures that the device has successfully received and processed the write request.

• Error Handling: Implement mechanisms to confirm that write operations are successful. Handle scenarios where writes fail, possibly due to connection issues or device unresponsiveness, by retrying or alerting the user.

5. Receiving and Parsing RPC Responses

Objective: Receive the response from the Shelly device, ensuring data integrity and correctness.

Process:

• Read Response Length from RX Control Characteristic: Begin by reading the response length from the RPC RX Control characteristic. This length indicates the size of the incoming response data, allowing the client to know how many bytes to expect.

• Unpack the Length: Convert the received 4-byte big-endian integer into an actual byte length value. This unpacked length determines the total size of the response data to be read.

• Read Response Data in Chunks: Due to BLE’s MTU (Maximum Transmission Unit) limitations, read the response data from the RPC Data characteristic in manageable chunks (typically 20 bytes). Continue reading until the entire response, as indicated by the response length, is received.

• Accumulate Response Data: As each chunk is read, append it to a buffer or byte array to reconstruct the complete response.

• Decode and Parse the Response: Once all chunks are received, decode the accumulated bytes into a UTF-8 string and parse the JSON object. This parsed response will contain either the result of the RPC call or an error field indicating a failure.

• Validate the Response: Ensure that the id in the response matches the original request id. This validation confirms that the response corresponds to the correct RPC request. Additionally, check for the presence of a result field to confirm successful execution or handle any error messages appropriately.

Considerations:

• Chunked Reads: BLE’s limited MTU size necessitates reading data in chunks to avoid truncation or data loss. Implement logic to handle partial reads and accumulate data until the complete response is received.

• Timeouts: Implement read timeouts to prevent indefinite waiting periods in case the Shelly device fails to respond or the connection is interrupted.

• Data Integrity: Validate that the entire response has been received and correctly parsed to ensure accurate and reliable communication.

Handling Common Challenges

Communicating with Shelly devices via BLE and RPC is a powerful approach, but it comes with its own set of challenges. Addressing these effectively ensures robust and reliable integrations.

1. MTU Size Management

Issue: BLE’s Maximum Transmission Unit (MTU) defines the maximum size of data packets that can be sent in a single transmission. Exceeding the MTU can lead to data truncation or failed transmissions.

Solution:

• Chunked Reads/Writes: Implement logic to split large data payloads into smaller chunks that adhere to the MTU size (typically 20 bytes). This ensures that data is transmitted reliably without exceeding BLE’s limitations.

Example Approach:

• Determine the total length of the data to be sent.

• Divide the data into chunks of 20 bytes or less.

• Sequentially write each chunk to the appropriate characteristic.

• Dynamic MTU Adjustment: Some BLE libraries and devices support negotiating a larger MTU size. If supported, adjust the MTU to optimize data transmission efficiency.

Example Consideration:

• Before initiating data transmission, request a larger MTU size if the library and device support it.

• Handle scenarios where the negotiation fails by reverting to standard chunk sizes.

Best Practices:

• Determine MTU Capability: Verify if both the Shelly device and the BLE client support larger MTUs to maximize data throughput.

• Implement Fallback Mechanisms: In cases where dynamic MTU negotiation is unsuccessful, default to standard chunk sizes to maintain compatibility.

2. Timeouts and Retries

Issue: BLE connections can be unstable, leading to timeouts or dropped communications, especially in environments with interference or multiple connected devices.

Solution:

• Implement Timeouts: Set appropriate timeout durations for connection attempts, read/write operations, and data processing steps. This prevents the application from waiting indefinitely for responses.

Example Approach:

• Define maximum wait times for each operation.

• If an operation exceeds its timeout, handle it gracefully by retrying or notifying the user.

• Retry Logic: Incorporate retry mechanisms for transient failures, such as temporary connection drops or failed writes. Limit the number of retries to avoid infinite loops and unnecessary resource consumption.

Example Approach:

• After a failed operation, wait for a short period before attempting a retry.

• Implement exponential backoff strategies to reduce the frequency of retries over time.

Best Practices:

• Exponential Backoff: Gradually increase the wait time between retries to reduce the likelihood of repeated failures, especially in high-interference environments.

• User Notifications: Inform users of persistent failures after exhausting retry attempts, enabling manual intervention if necessary.

3. Error Handling

Issue: Various errors can occur during BLE communication, including characteristic unavailability, malformed responses, or RPC-specific errors.

Solution:

• Comprehensive Exception Handling: Implement robust error-catching mechanisms at each stage of the communication process. This includes handling BLE-related exceptions, JSON parsing errors, and RPC-specific issues.

Example Approach:

• Wrap critical operations in try-catch blocks.

• Log detailed error messages to facilitate debugging and issue resolution.

• Validation Checks: Perform thorough validation of responses to ensure data integrity. This includes matching response IDs with request IDs and verifying the presence of expected fields like result or error.

Example Approach:

• After receiving a response, check if the id matches the original request.

• Ensure that the response contains either a result field or an error field to determine the outcome.

Best Practices:

• Logging: Maintain detailed logs of all communication attempts, successes, and failures. This aids in troubleshooting and understanding the system’s behavior under various conditions.

• Graceful Degradation: In cases of non-critical errors, allow the system to continue functioning by handling failures gracefully without crashing or disrupting other operations.

Conclusion

Communicating with Shelly devices via BLE and RPC offers a powerful avenue for creating sophisticated smart home integrations and automation solutions. By understanding the BLE architecture, GATT services, and RPC mechanisms, developers can craft robust and efficient communication frameworks tailored to their specific needs. This guide has provided a detailed roadmap to establish seamless communication, empowering you to harness the full potential of Shelly’s smart capabilities across various programming languages and platforms.

We Value Your Feedback!

Thank you for taking the time to read our article! Was it helpful or interesting?

Your insights can help us improve. We’d be grateful for any feedback. If you have a moment,

please share it with us on the following e-mail:

Integration@shelly.com