How to snif full-duplex UART protocol between two nodes

[pozz]

Many times I need to snif and log on a file the activity between two
nodes that talk over a full-duplex UART (separate TX and RX lines).

Just an example, consider a host MCU that talks with a modem through AT
commands. It's a half-duplex protocol because the MCU sends the AT
command and the modem replies. However the modem is able to send URC
messages that can be emitted at any time.

I think the better approach is to use two different UART/USB converters,
one for each signal (of course, both signals goes to the RX inputs of
the converters).

What about the sofware on the PC? It should open two serial ports and
monitors both of them, joining together the messages on a single file. A
great feature would be to have a timestamp for each message, where a
message is "\r\n" delimited.

Is there something already available that I can use?

[Don Y]

Many times I need to snif and log on a file the activity between two nodes that
talk over a full-duplex UART (separate TX and RX lines).

To be clear, neither of the endpoints will be a PC (which could
host the monitoring software).

What about the sofware on the PC? It should open two serial ports and monitors
both of them, joining together the messages on a single file. A great feature
would be to have a timestamp for each message, where a message is "\r\n"
delimited.
Is there something already available that I can use?

There are several PC-hosted "serial port monitors" that are available.
Some for no dollars.

I've used <https://www.232analyzer.com/>, in the past -- it has a "monitor"
mode that can provide timestamps.

If timing is super critical, you may need to explore hardware solutions,
instead.

The simple way of doing this is to spawn two copies of the tool, on
the same hosting PC (so their notion of time is similar). Capture
each log to a separate file. Then, glue the two files together
and sort(1) based on the timestamps.

This doesn't rule out ambiguities where the precision of the
timestamp makes two transactions appear concurrent when, in
fact, they likely occurred in a particular order (but too
close in time for the tool to differentiate with its timestamp).

The big downside is tat it isn't a real-time solution; the two files
have to be processed afterwards in a "batch" operation.

[I doubt *this* tool would behave well if both instances were
directed to the same log file! Who knows how the data is cached
within the app, how often the fd is flushed to disk, etc.]

But, it's a cheap starting point to identify other issues that
you may find significant!

[Theo]

Post by pozz
Many times I need to snif and log on a file the activity between two
nodes that talk over a full-duplex UART (separate TX and RX lines).
Just an example, consider a host MCU that talks with a modem through AT
commands. It's a half-duplex protocol because the MCU sends the AT
command and the modem replies. However the modem is able to send URC
messages that can be emitted at any time.
I think the better approach is to use two different UART/USB converters,
one for each signal (of course, both signals goes to the RX inputs of
the converters).
What about the sofware on the PC? It should open two serial ports and
monitors both of them, joining together the messages on a single file. A
great feature would be to have a timestamp for each message, where a
message is "\r\n" delimited.
Is there something already available that I can use?

Not quite what you asked for, but I've used:
https://sigrok.org/wiki/Main_Page
as logic analyser software, which will decode UART signals. You can get
some very cheap modules that it works with, eg:
https://sigrok.org/wiki/Lcsoft_Mini_Board
(about $10 from the usual marketplaces, search for 'CY7C68013A',
'EZ-USB' or 'FX2LP')
if you aren't fussy about buffering. Sigrok also supports a wide range of
other logic analysers - perhaps you already have something.

The nice thing about the CY7C68013A chip over regular logic analysers is
that it's capable of realtime streaming. Assuming your PC's USB stack is up
to it and doesn't randomly pause (or have some other chatty USB device on
the bus stealing bandwidth) you can record for as long as your memory and
storage subsystem can keep up. Given it's 480Mbps USB2.0, that is probably
quite a long time.

Being a logic analyser the inputs are by definition synchronised and
timestamped relative to each other.

Theo

[Richard Damon]

Post by pozz
Many times I need to snif and log on a file the activity between two
nodes that talk over a full-duplex UART (separate TX and RX lines).
Just an example, consider a host MCU that talks with a modem through AT
commands. It's a half-duplex protocol because the MCU sends the AT
command and the modem replies. However the modem is able to send URC
messages that can be emitted at any time.
I think the better approach is to use two different UART/USB converters,
one for each signal (of course, both signals goes to the RX inputs of
the converters).
What about the sofware on the PC? It should open two serial ports and
monitors both of them, joining together the messages on a single file. A
great feature would be to have a timestamp for each message, where a
message is "\r\n" delimited.
Is there something already available that I can use?

As Theo says, there are a number of different "Protocal Analyszers" that
you just insert into the communications link and it records all of the
data sent between the units, with time stamps, for display on a PC.

They are very handy for debuging such links.

The main thing you need to make sure of is that the module understands
the voltage format of your link, be it "TTL", RS-232, RS-422, or RS-485,
or whatever you have. And that it can capture at the baud rate of your link.

If the link might be variable baud, you need a unit that captures like a
logic analyizer and then lets you process later, which is a bit more
complicated. You also want something like that if you need to check if
there are bit-level timing issues with the link.

[Paul Rubin]

Post by pozz
Is there something already available that I can use?

There is a traditional hack where you make or buy a Y cable that lets
you tap the serial i/o and listen to both directions from a PC. Then
just log the data going in and out. When I did it I used a simple
Python script and it was able to keep up with the device we were dealing
with (might have been 1200 bps, I don't remember). It's possible
something like that already exists for Wireshark.

[pozz]

Post by Paul Rubin

Post by pozz
Is there something already available that I can use?

There is a traditional hack where you make or buy a Y cable that lets
you tap the serial i/o and listen to both directions from a PC. Then
just log the data going in and out. When I did it I used a simple
Python script and it was able to keep up with the device we were dealing
with (might have been 1200 bps, I don't remember). It's possible
something like that already exists for Wireshark.

Do you mean "joining electrically" (AND for UART TTL levels) the two
signals and connect the result to the RX signal of a single serial port
on the PC?

It couldn't work in my case, because the protocol could be full-duplex,
so both nodes could transmit at the same time. The AND result would be
corrupted.

[Don Y]

Post by Paul Rubin

Post by pozz
Is there something already available that I can use?

There is a traditional hack where you make or buy a Y cable that lets
you tap the serial i/o and listen to both directions from a PC.  Then
just log the data going in and out.  When I did it I used a simple
Python script and it was able to keep up with the device we were dealing
with (might have been 1200 bps, I don't remember).  It's possible
something like that already exists for Wireshark.

Do you mean "joining electrically" (AND for UART TTL levels) the two signals
and connect the result to the RX signal of a single serial port on the PC?

No. That won't work as the two transmitters could be trying to drive the
line (that they think only *they* own!) to different potentials.

Instead, you end up with 4 connectors -- the original two connecting your
"two communicating devices". Plus, two more that each have a single
connection to the RxD inputs of two serial ports on your PC (i.e., the
TxD connections from those two PC ports go nowhere).

This is a kludge as you have two receivers on each signal -- the receivers
on each of your two devices PLUS the receivers on the two PC ports.
And, you've got an extra ground involved -- the PC (but that won't typically
cause problems except in some special cases).

Alternatively, you can route the data *through* the PC; device A talks to
PC's port 1. PC echoes the stream to device B (keeping a log of the
content). At the same time, device B talks to PC's port 2 and the PC
similarly echoes the stream to device A (logging it).

The risk, here, is that the PC can introduce a delay in the transmission
between A and B. Likely this is not important in the example you cited.
And, if the PC can't keep up (because other things are happening in the
PC, at the time), then you risk losing characters.

Finally, this approach can be harder to implement completely if the
comms take advantage of other signalling means (like BREAKs). The
PC would have to recognize these in the data stream and reproduce them
in the same manner and in the same relationship to the surrounding
character transmissions.

It couldn't work in my case, because the protocol could be full-duplex, so both
nodes could transmit at the same time. The AND result would be corrupted.

Start simple. Monitor *one* device's output. See if it presents the
information of interest to you in a form that you can easily recognize
(or *extract* if you have to handle a control and data channel
sharing the same stream).

Then, the other to determine that it contains what you want/expect.

Finally, use a pair of PC ports wired to the two different transmitters
and collect both streams simultaneously.

Note that if you don't have genuine serial ports (e.g., USB), there may be
some delays in the delivery of the observed data to the PC (e.g., buffering
in the device as well as in the stack in the PC). This could introduce
jitter in the timestamps. Just something to watch for. It *won't* alter
the order in which data from each transmitter arrives at the PC but
could affect the skew between the two devices... if large enough,
you may see replies to commands (in your log) before you see the commands!

[Theo]

Post by Don Y

Post by Paul Rubin

Post by pozz
Is there something already available that I can use?

There is a traditional hack where you make or buy a Y cable that lets
you tap the serial i/o and listen to both directions from a PC.  Then
just log the data going in and out.  When I did it I used a simple
Python script and it was able to keep up with the device we were dealing
with (might have been 1200 bps, I don't remember).  It's possible
something like that already exists for Wireshark.

Do you mean "joining electrically" (AND for UART TTL levels) the two signals
and connect the result to the RX signal of a single serial port on the PC?

No. That won't work as the two transmitters could be trying to drive the
line (that they think only *they* own!) to different potentials.

I think it could work with a 'diode-OR' (or open-collector) kind of a
arrangement: either end can pull the line to the active state, but otherwise
the line will fall back to the inactive state. The diodes prevent the line
being driven to opposite potentials.

It would only really work protocol-wise if you could be sure both sides
weren't talking at the same time. That might work if there was a
command/response protocol where you could be sure everything was
half-duplex. And you also wouldn't be able to tell which end a byte came
from, unless there was internal structure in the messages (like ASCII
commands with carriage returns on the end).

But could work as a kind of poor-man's sniffer if you had no better option.

Theo

[Don Y]

Post by Theo

Post by Don Y

Post by Paul Rubin

Post by pozz
Is there something already available that I can use?

There is a traditional hack where you make or buy a Y cable that lets
you tap the serial i/o and listen to both directions from a PC.  Then
just log the data going in and out.  When I did it I used a simple
Python script and it was able to keep up with the device we were dealing
with (might have been 1200 bps, I don't remember).  It's possible
something like that already exists for Wireshark.

Do you mean "joining electrically" (AND for UART TTL levels) the two signals
and connect the result to the RX signal of a single serial port on the PC?

No. That won't work as the two transmitters could be trying to drive the
line (that they think only *they* own!) to different potentials.

I think it could work with a 'diode-OR' (or open-collector) kind of a
arrangement: either end can pull the line to the active state, but otherwise
the line will fall back to the inactive state. The diodes prevent the line
being driven to opposite potentials.

The problem is that you can't guarantee that the link will be run
half-duplex.

Post by Theo
It would only really work protocol-wise if you could be sure both sides
weren't talking at the same time. That might work if there was a
command/response protocol where you could be sure everything was
half-duplex. And you also wouldn't be able to tell which end a byte came
from, unless there was internal structure in the messages (like ASCII
commands with carriage returns on the end).

.. assuming the reply doesn't get started *while* the "redundant fluff"
at the end of the command is still on the wire (e.g., if the only commands
are A, B, C -- each terminated by a CRLF -- then an implementation might
decide to act once it has received the A, B *or* C... without bothering
to wait for the CRLF.

The problem with all (?) EIA232 comms is that there really is no
(universally) defined protocol -- even the format of the characters
can be made to vary in real time!

[E.g., I've used BREAK and LONG_BREAK as out-of-band controls to
let me *hardware* reset the device at the other end of the line
(watch for the line to stay in a spacing condition for >> one
character time and yank on the RESET- signal when that happens).
If you want to store-and-forward that, will you reproduce the
exact same spacing condition?]

Post by Theo
But could work as a kind of poor-man's sniffer if you had no better option.

The implementation issue I see the OP facing will be getting two
"predictable" serial ports on the same *PC* host (or, whatever other
device he uses to host the logging software).

I have a bit of code that I use to piece together RPC "calls" with
their corresponding "returns" on an ethernet link. It's easy to
get confused thinking that THIS reply was for THAT request -- when,
in fact, it was for the one before (or after).

Given the amount of buffering in 16450-ish UARTs -- and likely more
in USB dongles (when you consider the USB stack as part of the
issue), can you ever be sure to get data from one port in any given
timing relationship to the data from another? Esp when the stack is
likely a black box to you?

At least with old (e.g., 6402) UARTs, the buffering was limited to
exactly what *you* did in software; you only had to worry about the
character potentially on the wire and the one queued up behind it
(or ahead of it, depending on your perspective).

Buffering is a PITA to work around. Hence the appeal of hardware
solutions; "I saw this before I saw that".

[Rick C]

Post by Paul Rubin

Post by pozz
Is there something already available that I can use?

There is a traditional hack where you make or buy a Y cable that lets
you tap the serial i/o and listen to both directions from a PC. Then
just log the data going in and out. When I did it I used a simple
Python script and it was able to keep up with the device we were dealing
with (might have been 1200 bps, I don't remember). It's possible
something like that already exists for Wireshark.

Since this involves two devices communicating and two serial ports, one to monitor the data in each direction, wouldn't this be an X cable? Or am I missing something?

[Paul Rubin]

Post by Rick C
Since this involves two devices communicating and two serial ports,
one to monitor the data in each direction, wouldn't this be an X
cable? Or am I missing something?

I guess you could call it an X cable. This has a diagram that looks
about right:

https://www.lammertbies.nl/comm/cable/rs-232-spy-monitor#full

I've never made one of these myself. I used one that a co-worker made,
and wrote the code to log the data. I said 1200 bps earlier, but I
think it was actually 9600 bps. The connected devices were a POS
terminal and a receipt printer.

[Brian Cockburn]

Post by Paul Rubin
https://www.lammertbies.nl/comm/cable/rs-232-spy-monitor#full

That's exactly what is required.

[Grant Edwards]

Post by pozz
Many times I need to snif and log on a file the activity between two
nodes that talk over a full-duplex UART (separate TX and RX lines).

If you have to do that "many times", then buy a serial protocol
analyzer widget that has hardware that has two capture ports and
timestamps every bit/byte.

I have and older version of this and it works brilliantly:

https://www.iftools.com/analyzer/msb-rs232/index.en.php

There are similar but cheaper products from other vendors, but if it's
something you need to do often, then it's worth a few hundred
dollars/euros to get a decent one.

Post by pozz
Just an example, consider a host MCU that talks with a modem through AT
commands. It's a half-duplex protocol because the MCU sends the AT
command and the modem replies. However the modem is able to send URC
messages that can be emitted at any time.
I think the better approach is to use two different UART/USB converters,
one for each signal (of course, both signals goes to the RX inputs of
the converters).

The latency/buffer in the USB adpaters is awful. Spend some money and
by a real serial analyzer.

Post by pozz
What about the sofware on the PC? It should open two serial ports and
monitors both of them, joining together the messages on a single file. A
great feature would be to have a timestamp for each message, where a
message is "\r\n" delimited.

[Clifford Heath]

Post by pozz
Many times I need to snif and log on a file the activity between two
nodes that talk over a full-duplex UART (separate TX and RX lines).
Just an example, consider a host MCU that talks with a modem through AT
commands. It's a half-duplex protocol because the MCU sends the AT
command and the modem replies. However the modem is able to send URC
messages that can be emitted at any time.
I think the better approach is to use two different UART/USB converters,
one for each signal (of course, both signals goes to the RX inputs of
the converters).
What about the sofware on the PC? It should open two serial ports and
monitors both of them, joining together the messages on a single file. A
great feature would be to have a timestamp for each message, where a
message is "\r\n" delimited.
Is there something already available that I can use?

You could fairly easily modify my "connect.c" to do this, works in Linux
or OS/X. It does two-way communication between a serial port and the
terminal console you run it from, similar to `cu`, minicom` and similar
programs - but this is very simple, you don't have to navigate a wealth
of features to make changes.

Change it to configure two serial ports and copy data between them,
logging data to a file in the same way the Unix program `script` does.
Then plug in two USB/RS232 dongles and connect your two devices.

<https://www.dropbox.com/s/q0tqsszqx0q5hf4/connect.c>

Clifford Heath.

[Andrew Smallshaw]

Post by pozz
Many times I need to snif and log on a file the activity between two
nodes that talk over a full-duplex UART (separate TX and RX lines).
Is there something already available that I can use?

Try searching for "RS232 analyser", "RS232 datascope" and similar.
A quick look found a number of possible candidates. I won't make
any recommendation myself as last time I needed something like this
I went back to an old DOS tool.

[Andrew Smallshaw]

Post by pozz
Many times I need to snif and log on a file the activity between two
nodes that talk over a full-duplex UART (separate TX and RX lines).
Is there something already available that I can use?

Try searching for "RS232 analyser", "RS232 datascope" and similar.
A quick look found a number of possible candidates. I won't make
any recommendation myself as last time I needed something like this
I went back to an old DOS tool.