Skip to content

MQTT Proxy allows MQTT clients to send messages to other messaging systems

License

Notifications You must be signed in to change notification settings

grepplabs/mqtt-proxy

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

41 Commits
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

mqtt-proxy

Work in progress

License: CC BY-NC-ND 4.0 Build Docker Hub Docker Pulls

MQTT Proxy allows MQTT clients to send messages to other messaging systems

Implementation status

Install binary release

  1. Download the latest Linux release

    TAG=$(curl --silent "https://api.github.com/repos/grepplabs/mqtt-proxy/releases/latest" | jq -r '.tag_name')
    curl -Ls https://github.com/grepplabs/mqtt-proxy/releases/download/${TAG}/mqtt-proxy-${TAG}-linux-amd64.tar.gz | tar xz
    
  2. Move the binary in to your PATH.

    sudo mv ./mqtt-proxy /usr/local/bin/mqtt-proxy
    

Build

build binary

make clean build

build docker image

make clean docker-build

Helm 3 chart

Deploy the Helm chart

git clone git@github.com:grepplabs/mqtt-proxy.git
helm install mqtt-proxy ./mqtt-proxy/charts/mqtt-proxy \
  --set image.tag=latest \
  --set image.repository=grepplabs/mqtt-proxy \
  --values  <(echo '{
        "extraArgs" : ["server","--mqtt.publisher.name=noop"]
    }')

Test

prerequisites

kafka publisher

  1. build and start-up test environment

    cd scripts/cp-kafka
    make build-up
    
  2. subscribe to Kafka topic

    docker exec -it broker kafka-console-consumer --bootstrap-server localhost:9092 --topic mqtt-test --property print.key=true --from-beginning
    
  3. publish messages using mosquitto client

    • proxy using Kafka PLAINTEXT listener
    docker exec -it mqtt-client mosquitto_pub -L mqtt://mqtt-proxy:1883/dummy -m "test qos 0" --repeat 1 -q 0
    docker exec -it mqtt-client mosquitto_pub -L mqtt://mqtt-proxy:1883/dummy -m "test qos 1" --repeat 1 -q 1
    docker exec -it mqtt-client mosquitto_pub -L mqtt://mqtt-proxy:1883/dummy -m "test qos 2" --repeat 1 -q 2
    docker exec -it mqtt-client mosquitto_pub -L mqtt://mqtt-proxy:1883/dummy -m "test qos 0 / v5" --repeat 1 -q 0 -V mqttv5
    docker exec -it mqtt-client mosquitto_pub -L mqtt://mqtt-proxy:1883/dummy -m "test qos 1 / v5" --repeat 1 -q 1 -V mqttv5
    docker exec -it mqtt-client mosquitto_pub -L mqtt://mqtt-proxy:1883/dummy -m "test qos 2 / v5" --repeat 1 -q 2 -V mqttv5
    
    • proxy using Kafka SSL listener
    docker exec -it mqtt-client mosquitto_pub -L mqtt://mqtt-proxy-ssl:1884/dummy -m "test qos 0" --repeat 1 -q 0
    docker exec -it mqtt-client mosquitto_pub -L mqtt://mqtt-proxy-ssl:1884/dummy -m "test qos 1" --repeat 1 -q 1
    docker exec -it mqtt-client mosquitto_pub -L mqtt://mqtt-proxy-ssl:1884/dummy -m "test qos 2" --repeat 1 -q 2
    
  4. check the prometheus metrics

    watch -c 'curl -s localhost:9090/metrics | grep mqtt | egrep -v '^#''
    

publish to Amazon MSK

  1. provision test MSK and EC2 running in podman 2 proxy containers

    cd scripts/msk
    make tf-apply
    
  2. create Kafka mqtt-test topic

  3. publish

    • container connects to MSK PLAINTEXT listener
    mosquitto_pub -m "on" -t "dummy" -k 20 -i mqtt-proxy.clientv --repeat 1 -q 1 -h <ec2-ip> -p 1883
    
    • container connects to MSK TLS listener
    mosquitto_pub -m "on" -t "dummy" -k 20 -i mqtt-proxy.clientv --repeat 1 -q 1 -h <ec2-ip> -p 1884
    

SQS publisher

  1. Create AWS SQS test1 and test2.fifo queues

  2. Build and start MQTT Proxy

    make build
    ./mqtt-proxy server \
      --mqtt.publisher.name=sqs \
      --mqtt.publisher.message-format=json \
      --mqtt.publisher.sqs.queue-mappings=test1='^dummy$' \
      --mqtt.publisher.sqs.default-queue=test2.fifo \
      --mqtt.publisher.sqs.aws-profile=admin-dev
    
  3. publish

    mosquitto_pub -L mqtt://localhost:1883/dummy -m "test qos 2" --repeat 1 -q 2
    

SNS publisher

  1. Create AWS SNS test1 and test2.fifo topics with some subscription

  2. Build and start MQTT Proxy

    make build
    ./mqtt-proxy server \
      --mqtt.publisher.name=sns \
      --mqtt.publisher.message-format=json \
      --mqtt.publisher.sns.topic-arn-mappings=arn:aws:sns:eu-central-1:123456789012:test1='^dummy$' \
      --mqtt.publisher.sns.default-topic-arn=arn:aws:sns:eu-central-1:123456789012:test2.fifo \
      --mqtt.publisher.sns.aws-profile=admin-dev
    
  3. publish

    mosquitto_pub -L mqtt://localhost:1883/dummy -m "test qos 2" --repeat 1 -q 2
    

RabbitMQ publisher

  1. Start rabbitmq and create test queue

    cd scripts/rabbitmq
    docker-compose up
    curl -i -u user:bitnami -H "content-type:application/json" -XPUT -d'{"durable":true}' http://localhost:15672/api/queues/%2f/test
    
  2. Build and start MQTT Proxy

    make build
    ./mqtt-proxy server \
      --mqtt.publisher.name=rabbitmq \
      --mqtt.publisher.rabbitmq.username=user \
      --mqtt.publisher.rabbitmq.password=bitnami \
      --mqtt.publisher.rabbitmq.default-queue=test \
      --mqtt.publisher.rabbitmq.confirms.exactly-once=true
    
  3. publish

    mosquitto_pub -L mqtt://localhost:1883/dummy -m "test qos 0" --repeat 1 -q 0
    

plain authenticator

  1. start server with plain authenticator

    • with credentials file
    cat <<EOF > mqtt-credentials.csv
    alice,alice-secret
    "bob","bob-secret"
    EOF
    
    mqtt-proxy server --mqtt.publisher.name=noop \
        --mqtt.handler.auth.name=plain \
        --mqtt.handler.auth.plain.credentials-file=mqtt-credentials.csv
    
    • providing credentials as parameters
    mqtt-proxy server --mqtt.publisher.name=noop \
        --mqtt.handler.auth.name=plain \
        --mqtt.handler.auth.plain.credentials=alice=alice-secret \
        --mqtt.handler.auth.plain.credentials=bob=bob-secret
    
  2. publish

    mosquitto_pub -m "on" -t "dummy" -u alice -P alice-secret
    mosquitto_pub -L mqtt://bob:bob-secret@localhost:1883/dummy -m "on"
    

Configuration

Kafka publisher

Kafka producer configuration properties used by librdkafka should be prefixed with producer.

--mqtt.publisher.kafka.config=producer.sasl.mechanisms=PLAIN,producer.security.protocol=SASL_SSL,producer.sasl.username=myuser,producer.sasl.password=mypasswd

Examples

  • Ignore subscribe / unsubscribe requests
mqtt-proxy server  --mqtt.publisher.name=noop --mqtt.handler.ignore-unsupported SUBSCRIBE --mqtt.handler.ignore-unsupported UNSUBSCRIBE

Metrics

metric labels description
mqtt_proxy_build_info branch, goversion, revision, revision A metric with a constant '1' value labeled by version, revision, branch, and goversion from which mqtt_proxy was built.
mqtt_proxy_server_connections_active Number of active TCP connections from clients to server.
mqtt_proxy_server_connections_total Total number of TCP connections from clients to server.
mqtt_proxy_handler_requests_total type, version Total number of MQTT requests labeled by package control type and protocol version.
mqtt_proxy_handler_responses_total type, version Total number of MQTT responses labeled by package control type and protocol version.
mqtt_proxy_publisher_publish_duration_seconds name, type, qos Histogram tracking latencies for publish requests.
mqtt_proxy_authenticator_login_duration_seconds name, code, err Histogram tracking latencies for login requests.