Arduino pipeline

mariona · 31 May 2023 14:44

Hi!
I’m doing a project with Quix, and I was wondering if i can make the pipeline the data of information be one project of arduino. Could that be possible? I would need to make a pipeline with Kafka or another service?
Thank you and 'm really looking forward to hearing the response on the forum!

merltron · 31 May 2023 15:09

Hi Mariona, welcome!

Can you explain a bit more about what you want to do? For example what kind of data would you like to process?

Without understanding the full use case, I could suggest that you follow the first few steps in this tutorial but replace all the “big data” stuff with Quix.

I’m not sure you even need Apache NiFi, but I’ll let a more knowledgeable colleague chime in here.

In any case, you can then use our MQTT connector to get data into Quix-hosted Kafka.
Here’s more info about that connector: quix-samples/python/sources/MQTT at main · quixio/quix-samples · GitHub

Hope that helps!

Steve · 31 May 2023 15:14

Hi @mariona, In addition to Merlins answer and assuming you are using an Arduino board with wifi capability you could try posting your data to our streaming writer API.

Documentation for this is here: Introduction - Quix Docs

mariona · 31 May 2023 15:19

Hii thanks for the quick reply!
I need to process data of 2 sensors, one of one button and one of distance, but in the code of Arduino it gives me the response of the state, like a message, I would like to put this messages in a pipeline in quix and after connect it to a Slack notification.
The problem is that i dont know how to connect the messages of arduino in the pipeline.

mariona · 31 May 2023 15:23

Hi Steve ! No, unfortunately I use an Arduino UNO digital, with no connection of Wi-Fi. Is necessary to have a connection Wi-Fi to make the connection of the data?

Steve · 31 May 2023 15:42

Hi, you will need to get the data out of the Arduino somehow.

They make an UNO WIFI… https://docs.arduino.cc/retired/boards/arduino-uno-wifi
And I have used some esp8266 boards that have wifi on them.

If not you will have to transmit the data via some other method to get it “online”.
This will be the same for any platform you use not just Quix.

Is your board connected to a laptop or PC? Could you transmit the data over USB and then publish from the laptop to Quix?

Or perhaps you have some other radio like a 433mhz module or bluetooth module that you can use to transmit data to a PC or phone?

mariona · 31 May 2023 16:00

Hi steve!
Yes its connected to my laptop via usb, that would be the problem, how can i publish the data i recive in quix? In kafka?

Steve · 1 June 2023 06:54

Hi.

If you can get the data to your PC then you can use either Python, C# or even node.js to publish the data.

I suggest you have a look at the getting started guide here: Quickstart - Quix Docs

You’ll want to create a producer sine you’ll be publishing data into Quix.

sadeep · 1 June 2023 09:10

Hi Mariona,

That’s a very cool project you’re working on. To get data from an Arduino Uno, you have two options. One is to use the serial interface of Arduino to send data to your computer. Then use a library like pyserial to read that data from your computer. Then you can use our Quix Streams Python library to stream your sensor data to Quix.

While the above suggestion will work during development and testing, the downside is you need to have a PC next to your Arduino for that to work. When I want to do something like this, I tend to use a WiFi shield like ESP8266 to give Arduino Uno internet access. You can program ESP chips using the Arduino IDE. So, in this case, connect the ESP chip to Arduino Uno’s serial (RX and TX), and use the Serial interface to send sensor data to the ESP chip. Then, you can publish that data to Quix using our web API. This is the best way to get your data to Quix without a PC close by. We also support MQTT, so you could publish data to somewhere like HiveMQ and then connect Quix to it. There’s an MQTT connector on Quix you can use for the latter approach. ESP has libraries for both HTTP and MQTT communications.

For now, I recommend just sending data to your PC over Serial and using Python (Pyserial) to read that data and re-stream to Quix to get things working without having to buy new parts

mariona · 1 June 2023 11:59

Thanks!
Yeah im just doing that but when I execute de code you gave me, it gives me problems with the package of quixstreamclient.py, of the library of quixstreams, its a problem of my code or the library its not download correcty?

  File "C:\Users\Usuario\Desktop\data_stream.py", line 13, in <module>
    topic = client.get_topic_producer(topic_name)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Usuario\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\quixstreams\quixstreamingclient.py", line 181, in get_topic_producer
    dotnet_pointer = self._interop.GetTopicProducer(topic_id_or_name)
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Usuario\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\quixstreams\native\Python\QuixStreamsStreaming\QuixStreamingClient.py", line 283, in GetTopicProducer
    result = InteropUtils.invoke("quixstreamingclient_gettopicproducer", self._pointer, topicIdOrName_ptr)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Usuario\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\quixstreams\native\Python\InteropHelpers\InteropUtils.py", line 86, in invoke
    InteropUtils.__exception_handler(ex)
  File "C:\Users\Usuario\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\quixstreams\helpers\exceptionconverter.py", line 10, in raise_from_interop
    raise QuixApiException(exception.message)

I dont know if its the code or something is missing, thanks for the help

sadeep · 1 June 2023 13:46

Hi Mariona,

Can you please post the Python code (redact credentials and any sensitive configuration values), so I can try to reproduce this. Also, I assume what you posted is the full error message? Otherwise, make sure you post the full stack trace (redacting sensitive information, if there’s any). Thanks.

mariona · 1 June 2023 14:15

Thank you for helping me ,
The code i actually have is this, maybe is an error because im executing in windows?

import serial
import quixstreams as qx
from quixstreams import EventData
import pandas as pd
import time
import datetime

quix_account_token = "" //the quix token
topic_name = "" //my topic
client = qx.QuixStreamingClient(quix_account_token)
topic = client.get_topic_producer(topic_name)
stream = topic.create_stream()

with serial.Serial('COM5', 9600, timeout=10) as ser:
	t=serialreadline()
	data = qx.TimeSeriesData()
	stream.events.publish(data)

This is the full stack trace, it gives me that im not a user in quix?
But the token I used its copied directy form quix.io

  File "C:\Users\Usuario\Desktop\data_stream.py", line 14, in <module>
    topic = client.get_topic_producer(topic_name)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Usuario\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\quixstreams\quixstreamingclient.py", line 181, in get_topic_producer
    dotnet_pointer = self._interop.GetTopicProducer(topic_id_or_name)
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Usuario\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\quixstreams\native\Python\QuixStreamsStreaming\QuixStreamingClient.py", line 283, in GetTopicProducer
    result = InteropUtils.invoke("quixstreamingclient_gettopicproducer", self._pointer, topicIdOrName_ptr)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Usuario\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\quixstreams\native\Python\InteropHelpers\InteropUtils.py", line 86, in invoke
    InteropUtils.__exception_handler(ex)
  File "C:\Users\Usuario\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\quixstreams\helpers\exceptionconverter.py", line 10, in raise_from_interop
    raise QuixApiException(exception.message)
quixstreams.exceptions.quixapiexception.QuixApiException: Request failed (401) to https://portal-api.platform.quix.ai/workspaces with message: User is not authenticated. If you need help, contact us with Correlation Id .

Thankss again

mariona · 1 June 2023 15:20

Hi @sadeep i just fix it! It was I had another user token of quix.io in the code that was not correct, but now i have problems with the port.
This is the error now:

C:\Users\Usuario\Desktop>python3 data_stream.py
Traceback (most recent call last):
  File "C:\Users\Usuario\Desktop\data_stream.py", line 15, in <module>
    with serial.Serial('COM5', 9600, timeout=10) as ser:
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Usuario\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\serial\serialwin32.py", line 33, in __init__
    super(Serial, self).__init__(*args, **kwargs)
  File "C:\Users\Usuario\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\serial\serialutil.py", line 244, in __init__
    self.open()
  File "C:\Users\Usuario\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\serial\serialwin32.py", line 64, in open
    raise SerialException("could not open port {!r}: {!r}".format(self.portstr, ctypes.WinError()))
serial.serialutil.SerialException: could not open port 'COM5': FileNotFoundError(2, 'El sistema no puede encontrar el archivo especificado.', None, 2)

It doesnt read the port arduino is connected in and I dont know why

sadeep · 2 June 2023 08:13

Hi Mariona,

Try com5 instead of COM5. I’m not very familiar with serial ports on Windows, but the error indicates that ‘COM5’ is not the port your Arduino is connected to. So, either the port changed at some point (after disconnecting the Arduino perhaps?) or the port name is misspelled.

mariona · 2 June 2023 08:56

Hi!
I think is more of a problem in the permissions of windows and the ports, because in arduino it shows that the port COM5 is the board connected to it:

I changed the code a little bit, but is continues to give me the same error:

import serial
import quixstreams as qx
from quixstreams import EventData
import pandas as pd
import socket
import time
import datetime

quix_account_token = "" //my token
topic_name = "" //my sensors
client = qx.QuixStreamingClient(quix_account_token)
topic = client.get_topic_producer(topic_name)
stream = topic.create_stream()

with serial.Serial('COM5', 9600, timeout=10) as ser:
	line = ser.readline().decode().strip()
data = qx.TimeSeriesData()
data["timestamp"] = datetime.datetime.now()
data["value"] = line
stream.events.publish(data)

The error in question:

C:\Users\Usuario\Desktop> python3 data_stream.py
Traceback (most recent call last):
  File "C:\Users\Usuario\Desktop\data_stream.py", line 15, in <module>
    with serial.Serial('COM5', 9600, timeout=10) as ser:
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Usuario\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\serial\serialwin32.py", line 33, in __init__
    super(Serial, self).__init__(*args, **kwargs)
  File "C:\Users\Usuario\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\serial\serialutil.py", line 244, in __init__
    self.open()
  File "C:\Users\Usuario\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\serial\serialwin32.py", line 64, in open
    raise SerialException("could not open port {!r}: {!r}".format(self.portstr, ctypes.WinError()))
serial.serialutil.SerialException: could not open port 'COM5': PermissionError(13, 'Acceso denegado.', None, 5)

Could i need to change something of the permissions? If so do you know how?
Thanks for the help

sadeep · 6 June 2023 13:50

Hi Mariona,

When you are running your Python program, make sure you close the serial monitor of the Arduino IDE. Only one program should be accessing the serial port at a time. Also, you might want to check StackOverflow for more help on serial communication.