|
| 1 | +# What is EMQ X ? |
| 2 | + |
| 3 | +[EMQ X MQTT broker](https://www.emqx.io/) is a fully open source, highly scalable, highly available distributed MQTT messaging broker for IoT, M2M and Mobile applications that can handle tens of millions of concurrent clients. |
| 4 | + |
| 5 | +Starting from 3.0 release, *EMQ X* broker fully supports MQTT V5.0 protocol specifications and backward compatible with MQTT V3.1 and V3.1.1, as well as other communication protocols such as MQTT-SN, CoAP, LwM2M, WebSocket and STOMP. The 3.0 release of the *EMQ X* broker can scaled to 10+ million concurrent MQTT connections on one cluster. |
| 6 | + |
| 7 | +%%LOGO%% |
| 8 | + |
| 9 | +# How to use this image |
| 10 | + |
| 11 | +### Run emqx |
| 12 | + |
| 13 | +Execute some command under this docker image |
| 14 | + |
| 15 | +```console |
| 16 | +$ docker run -d --name emqx %%IMAGE%%:tag |
| 17 | +``` |
| 18 | + |
| 19 | +For example |
| 20 | + |
| 21 | +```console |
| 22 | +$ docker run -d --name emqx -p 18083:18083 -p 1883:1883 %%IMAGE%%:latest |
| 23 | +``` |
| 24 | + |
| 25 | +The emqx broker runs as linux user `emqx` in the docker container. |
| 26 | + |
| 27 | +### Configuration |
| 28 | + |
| 29 | +Use the environment variable to configure the EMQ X docker container. |
| 30 | + |
| 31 | +By default, the environment variables with `EMQX_` prefix are mapped to key-value pairs in configuration files. |
| 32 | + |
| 33 | +You can change the prefix by overriding `HOCON_ENV_OVERRIDE_PREFIX`. |
| 34 | + |
| 35 | +Example: |
| 36 | + |
| 37 | +```bash |
| 38 | +EMQX_LISTENERS__SSL__DEFAULT__ACCEPTORS <--> listeners.ssl.default.acceptors |
| 39 | +EMQX_ZONES__DEFAULT__MQTT__MAX_PACKET_SIZE <--> zones.default.mqtt.max_packet_size |
| 40 | +``` |
| 41 | + |
| 42 | +- Prefix `EMQX_` is removed |
| 43 | +- All upper case letters is replaced with lower case letters |
| 44 | +- `__` is replaced with `.` |
| 45 | + |
| 46 | +If `HOCON_ENV_OVERRIDE_PREFIX=DEV_` is set: |
| 47 | + |
| 48 | +```bash |
| 49 | +DEV_LISTENER__SSL__EXTERNAL__ACCEPTORS <--> listener.ssl.external.acceptors |
| 50 | +DEV_MQTT__MAX_PACKET_SIZE <--> mqtt.max_packet_size |
| 51 | +``` |
| 52 | + |
| 53 | +Non mapped environment variables: |
| 54 | + |
| 55 | +```bash |
| 56 | +EMQX_NAME |
| 57 | +EMQX_HOST |
| 58 | +``` |
| 59 | + |
| 60 | +These environment variables will ignore for configuration file. |
| 61 | + |
| 62 | +#### EMQ X Configuration |
| 63 | + |
| 64 | +> NOTE: All EMQ X Configuration in [etc/emqx.conf](https://github.com/emqx/emqx/blob/master/etc/emqx.conf) could config by environment. The following list is just an example, not a complete configuration. |
| 65 | +
|
| 66 | +| Options | Default | Mapped | Description | |
| 67 | +|-------------|----------------|--------|----------------------------| |
| 68 | +| `EMQX_NAME` | container name | none | emqx node short name | |
| 69 | +| `EMQX_HOST` | container IP | none | emqx node host, IP or FQDN | |
| 70 | + |
| 71 | +The list is incomplete and may changed with [etc/emqx.conf](https://github.com/emqx/emqx/blob/master/etc/emqx.conf) and plugin configuration files. But the mapping rule is similar. |
| 72 | + |
| 73 | +If set `EMQX_NAME` and `EMQX_HOST`, and unset `EMQX_NODE_NAME`, `EMQX_NODE_NAME=$EMQX_NAME@$EMQX_HOST`. |
| 74 | + |
| 75 | +For example, set mqtt tcp port to 1883 |
| 76 | + |
| 77 | +```console |
| 78 | +$ docker run -d --name emqx -e EMQX__LISTENERS__TCP__DEFAULT__BIND=1883 -p 18083:18083 -p 1883:1883 %%IMAGE%%:latest |
| 79 | +``` |
| 80 | + |
| 81 | +#### EMQ Loaded Modules Configuration |
| 82 | + |
| 83 | +| Options | Default | Description | |
| 84 | +|-----------------------|-------------------|-----------------------------| |
| 85 | +| `EMQX_LOADED_MODULES` | see content below | default modules emqx loaded | |
| 86 | + |
| 87 | +Default environment variable `EMQX_LOADED_MODULES`, including |
| 88 | + |
| 89 | +- `emqx_mod_presence` |
| 90 | + |
| 91 | +```bash |
| 92 | +# The default EMQX_LOADED_MODULES env |
| 93 | +EMQX_LOADED_MODULES="emqx_mod_presence" |
| 94 | +``` |
| 95 | + |
| 96 | +For example, set `EMQX_LOADED_MODULES=emqx_mod_delayed,emqx_mod_rewrite` to load these two modules. |
| 97 | + |
| 98 | +You can use comma, space or other separator that you want. |
| 99 | + |
| 100 | +All the modules defined in env `EMQX_LOADED_MODULES` will be loaded. |
| 101 | + |
| 102 | +```bash |
| 103 | +EMQX_LOADED_MODULES="emqx_mod_delayed,emqx_mod_rewrite" |
| 104 | +EMQX_LOADED_MODULES="emqx_mod_delayed emqx_mod_rewrite" |
| 105 | +EMQX_LOADED_MODULES="emqx_mod_delayed | emqx_mod_rewrite" |
| 106 | +``` |
| 107 | + |
| 108 | +#### EMQ Loaded Plugins Configuration |
| 109 | + |
| 110 | +| Options | Default | Description | |
| 111 | +|-----------------------|-------------------|-----------------------------| |
| 112 | +| `EMQX_LOADED_PLUGINS` | see content below | default plugins emqx loaded | |
| 113 | + |
| 114 | +Default environment variable `EMQX_LOADED_PLUGINS`, including |
| 115 | + |
| 116 | +- `emqx_recon` |
| 117 | +- `emqx_retainer` |
| 118 | +- `emqx_rule_engine` |
| 119 | +- `emqx_management` |
| 120 | +- `emqx_dashboard` |
| 121 | + |
| 122 | +```bash |
| 123 | +# The default EMQX_LOADED_PLUGINS env |
| 124 | +EMQX_LOADED_PLUGINS="emqx_recon,emqx_retainer,emqx_management,emqx_dashboard" |
| 125 | +``` |
| 126 | + |
| 127 | +For example, set `EMQX_LOADED_PLUGINS= emqx_retainer,emqx_rule_engine` to load these two plugins. |
| 128 | + |
| 129 | +You can use comma, space or other separator that you want. |
| 130 | + |
| 131 | +All the plugins defined in `EMQX_LOADED_PLUGINS` will be loaded. |
| 132 | + |
| 133 | +```bash |
| 134 | +EMQX_LOADED_PLUGINS="emqx_retainer,emqx_rule_engine" |
| 135 | +EMQX_LOADED_PLUGINS="emqx_retainer emqx_rule_engine" |
| 136 | +EMQX_LOADED_PLUGINS="emqx_retainer | emqx_rule_engine" |
| 137 | +``` |
| 138 | + |
| 139 | +#### EMQ X Plugins Configuration |
| 140 | + |
| 141 | +The environment variables which with `EMQX_` prefix are mapped to all emqx plugins' configuration file, `.` get replaced by `__`. |
| 142 | + |
| 143 | +Example: |
| 144 | + |
| 145 | +```bash |
| 146 | +EMQX_RETAINER__STORAGE_TYPE <--> retainer.storage_type |
| 147 | +EMQX_RETAINER__MAX_PAYLOAD_SIZE <--> retainer.max_payload_size |
| 148 | +``` |
| 149 | + |
| 150 | +Don't worry about where to find the configuration file of emqx plugins, this docker image will find and config them automatically using some magic. |
| 151 | + |
| 152 | +All plugin of emqx project could config in this way, following the environment variables mapping rule above. |
| 153 | + |
| 154 | +Assume you are using redis auth plugin, for example: |
| 155 | + |
| 156 | +```bash |
| 157 | +#EMQX_RETAINER__STORAGE_TYPE = "ram" |
| 158 | +#EMQX_RETAINER.MAX_PAYLOAD_SIZE = 1MB |
| 159 | + |
| 160 | +docker run -d --name emqx -p 18083:18083 -p 1883:1883 -p 4369:4369 \ |
| 161 | + -e EMQX_LISTENERS__TCP__DEFAULT=1883 \ |
| 162 | + -e EMQX_LOADED_PLUGINS="emqx_retainer" \ |
| 163 | + -e EMQX_RETAINER__STORAGE_TYPE = "ram" \ |
| 164 | + -e EMQX_RETAINER__MAX_PAYLOAD_SIZE = 1MB \ |
| 165 | + %%IMAGE%%:latest |
| 166 | +``` |
| 167 | + |
| 168 | +For numbered configuration options where the number is next to a `.` such as: |
| 169 | + |
| 170 | +- backend.redis.pool1.server |
| 171 | +- backend.redis.hook.message.publish.1 |
| 172 | + |
| 173 | +You can configure an arbitrary number of them as long as each has a uniq unber for it's own configuration option: |
| 174 | + |
| 175 | +```bash |
| 176 | +docker run -d --name emqx -p 18083:18083 -p 1883:1883 -p 4369:4369 \ |
| 177 | + -e EMQX_BACKEND_REDIS_POOL1__SERVER=127.0.0.1:6379 \ |
| 178 | + [...] |
| 179 | + -e EMQX_BACKEND__REDIS__POOL5__SERVER=127.0.0.5:6379 \ |
| 180 | + -e EMQX_BACKEND__REDIS__HOOK_MESSAGE__PUBLISH__1='{"topic": "persistant/topic1", "action": {"function": "on_message_publish"}, "pool": "pool1"}' \ |
| 181 | + -e EMQX_BACKEND__REDIS__HOOK_MESSAGE__PUBLISH__2='{"topic": "persistant/topic2", "action": {"function": "on_message_publish"}, "pool": "pool1"}' \ |
| 182 | + -e EMQX_BACKEND__REDIS__HOOK_MESSAGE__PUBLISH__3='{"topic": "persistant/topic3", "action": {"function": "on_message_publish"}, "pool": "pool1"}' \ |
| 183 | + [...] |
| 184 | + -e EMQX_BACKEND__REDIS__HOOK_MESSAGE__PUBLISH__13='{"topic": "persistant/topic13", "action": {"function": "on_message_publish"}, "pool": "pool1"}' \ |
| 185 | + %%IMAGE%%:latest |
| 186 | +``` |
| 187 | + |
| 188 | +### Cluster |
| 189 | + |
| 190 | +EMQ X supports a variety of clustering methods, see our [documentation](https://docs.emqx.io/broker/latest/en/advanced/cluster.html#emqx-service-discovery) for details. |
| 191 | + |
| 192 | +Let's create a static node list cluster from docker-compose. |
| 193 | + |
| 194 | +- Create `docker-compose.yaml`: |
| 195 | + |
| 196 | +```yaml |
| 197 | + version: '3' |
| 198 | + |
| 199 | + services: |
| 200 | + emqx1: |
| 201 | + image: %%IMAGE%%:latest |
| 202 | + environment: |
| 203 | + - "EMQX_NAME=emqx" |
| 204 | + - "EMQX_HOST=node1.emqx.io" |
| 205 | + - "EMQX_CLUSTER__DISCOVERY_STRATEGY=static" |
| 206 | + - "EMQX_CLUSTER__STATIC__SEEDS=[[email protected], [email protected]]" |
| 207 | + networks: |
| 208 | + emqx-bridge: |
| 209 | + aliases: |
| 210 | + - node1.emqx.io |
| 211 | + |
| 212 | + emqx2: |
| 213 | + image: %%IMAGE%%:latest |
| 214 | + environment: |
| 215 | + - "EMQX_NAME=emqx" |
| 216 | + - "EMQX_HOST=node2.emqx.io" |
| 217 | + - "EMQX_CLUSTER__DISCOVERY_STRATEGY=static" |
| 218 | + - "EMQX_CLUSTER__STATIC__SEEDS=[[email protected], [email protected]]" |
| 219 | + networks: |
| 220 | + emqx-bridge: |
| 221 | + aliases: |
| 222 | + - node2.emqx.io |
| 223 | + |
| 224 | + networks: |
| 225 | + emqx-bridge: |
| 226 | + driver: bridge |
| 227 | +``` |
| 228 | +
|
| 229 | +- Start the docker-compose cluster |
| 230 | +
|
| 231 | +```bash |
| 232 | + docker-compose -p my_emqx up -d |
| 233 | +``` |
| 234 | + |
| 235 | +- View cluster |
| 236 | + |
| 237 | +```bash |
| 238 | + $ docker exec -it my_emqx_emqx1_1 sh -c "emqx_ctl cluster status" |
| 239 | + Cluster status: #{running_nodes = > [ '[email protected]', '[email protected]'], |
| 240 | + stopped_nodes => []} |
| 241 | +``` |
| 242 | + |
| 243 | +### Persistence |
| 244 | + |
| 245 | +If you want to persist the EMQ X docker container, you need to keep the following directories: |
| 246 | + |
| 247 | +- `/opt/emqx/data` |
| 248 | +- `/opt/emqx/etc` |
| 249 | +- `/opt/emqx/log` |
| 250 | + |
| 251 | +Since data in these folders are partially stored under the `/opt/emqx/data/mnesia/${node_name}`, the user also needs to reuse the same node name to see the previous state. In detail, one needs to specify the two environment variables: `EMQX_NAME` and `EMQX_HOST`, `EMQX_HOST` set as `127.0.0.1` or network alias would be useful. |
| 252 | + |
| 253 | +In if you use docker-compose, the configuration would look something like this: |
| 254 | + |
| 255 | +```YAML |
| 256 | +volumes: |
| 257 | + vol-emqx-data: |
| 258 | + name: foo-emqx-data |
| 259 | + vol-emqx-etc: |
| 260 | + name: foo-emqx-etc |
| 261 | + vol-emqx-log: |
| 262 | + name: foo-emqx-log |
| 263 | + |
| 264 | +services: |
| 265 | + emqx: |
| 266 | + image: %%IMAGE%%:latest |
| 267 | + restart: always |
| 268 | + environment: |
| 269 | + EMQX_NAME: foo_emqx |
| 270 | + EMQX_HOST: 127.0.0.1 |
| 271 | + volumes: |
| 272 | + - vol-emqx-data:/opt/emqx/data |
| 273 | + |
| 274 | + - vol-emqx-log:/opt/emqx/log |
| 275 | +``` |
| 276 | +
|
| 277 | +### Kernel Tuning |
| 278 | +
|
| 279 | +Under linux host machine, the easiest way is [Tuning guide](https://docs.emqx.io/en/broker/latest/tutorial/tune.html#linux-kernel-tuning). |
| 280 | +
|
| 281 | +If you want tune linux kernel by docker, you must ensure your docker is latest version (>=1.12). |
| 282 | +
|
| 283 | +```bash |
| 284 | + |
| 285 | +docker run -d --name emqx -p 18083:18083 -p 1883:1883 -p 4369:4369 \ |
| 286 | + --sysctl fs.file-max=2097152 \ |
| 287 | + --sysctl fs.nr_open=2097152 \ |
| 288 | + --sysctl net.core.somaxconn=32768 \ |
| 289 | + --sysctl net.ipv4.tcp_max_syn_backlog=16384 \ |
| 290 | + --sysctl net.core.netdev_max_backlog=16384 \ |
| 291 | + --sysctl net.ipv4.ip_local_port_range=1000 65535 \ |
| 292 | + --sysctl net.core.rmem_default=262144 \ |
| 293 | + --sysctl net.core.wmem_default=262144 \ |
| 294 | + --sysctl net.core.rmem_max=16777216 \ |
| 295 | + --sysctl net.core.wmem_max=16777216 \ |
| 296 | + --sysctl net.core.optmem_max=16777216 \ |
| 297 | + --sysctl net.ipv4.tcp_rmem=1024 4096 16777216 \ |
| 298 | + --sysctl net.ipv4.tcp_wmem=1024 4096 16777216 \ |
| 299 | + --sysctl net.ipv4.tcp_max_tw_buckets=1048576 \ |
| 300 | + --sysctl net.ipv4.tcp_fin_timeout=15 \ |
| 301 | + %%IMAGE%%:latest |
| 302 | + |
| 303 | +``` |
| 304 | + |
| 305 | +> REMEMBER: DO NOT RUN EMQ X DOCKER PRIVILEGED OR MOUNT SYSTEM PROC IN CONTAINER TO TUNE LINUX KERNEL, IT IS UNSAFE. |
0 commit comments