CRC16 - Modbus CRC Calculation

[Tangerino]

Is there any crc16 function available to replace the CRC calculation in Python for ESP32?

[mendenm]

@Tangerino

I found this, which might help...

https://www.digi.com/resources/documentation/digidocs/90001537/references/r_python_crc16_modbus.htm

along with a more detailed discussion of the process:

https://stackoverflow.com/questions/55179867/crc16-modbus-computing-algorithm

[Tangerino]

My question was totally incomplete, I mean a native C function to replace the python one.
Sorry and thank you

[andrewleech]

I've used viper to get near-c performance out of a python crc function previously, this is much easier to integrate / distribute than a c module typically. Here's an example (not crc16) that might be helpful:

@micropython.viper()
def crc32_xfer(data: ptr8, size: int) -> uint:
    """
    Returns the 32 bit CRC (XFER algorithm) of the provided bytes

    Compatible with CRC-32/XFER from https://crccalc.com/?method=crc32
    If the returned crc is appended to the original data and run back through
    this function the result will be 0.
    :param bytes,bytearray data: bytes to calculate crc on
    :return int: crc of input data
    """
    _poly:uint = 0x000000AF
    crc = uint(0)
    highbit = int(1 << (32 - 1))
    mask = int((highbit - 1) << 1) | 0x1
    poly = _poly
    shift = 32 - 8
    diff8 = -1 * shift
    if diff8 > 0:
        mask = 0xFF
        crc <<= diff8
        shift = 0
        highbit = 0x80
        poly = _poly << diff8

    for i in range(size):
        crc ^= data[i] << shift
        for i in range(8):
            if crc & highbit:
                crc = (crc << 1) ^ poly
            else:
                crc = (crc << 1)
        crc &= mask
    if diff8 > 0:
        crc >>= diff8
    return crc

[rkompass]

Last December I did a CRC library that contains viper, asm_thumb and asm_xtensa implementations.
I just uploaded it here.

It has definitions for many 8, 16, 32 and 64 bit CRC variants and is as fast as a table-based implementation using these accelerators can be. There is also a benchmark.
Unfortunately for the esp32 the @micropython.asm_xtensawin decorator would be needed, which is not available. So you are limited to viper if you want to do fast calculations (without resorting to C) on esp32.

Anyone interested in working on adding the xtensawin variant of the xtensa assembler? Which would mainly be dealing with the complexities of the windowing in it, I suppose.

[stephanelsmith]

This is great! What a great resource.

[stephanelsmith]

Lots of great resources out there. I've transcribed two from c, references included.

Here is a crc16 (xmodem) version:

#https://code.google.com/archive/p/pycrc16/
CRC16_XMODEM_TABLE = [
        0x0000, 0x1021, 0x2042, 0x3063, 0x4084, 0x50a5, 0x60c6, 0x70e7,
        0x8108, 0x9129, 0xa14a, 0xb16b, 0xc18c, 0xd1ad, 0xe1ce, 0xf1ef,
        0x1231, 0x0210, 0x3273, 0x2252, 0x52b5, 0x4294, 0x72f7, 0x62d6,
        0x9339, 0x8318, 0xb37b, 0xa35a, 0xd3bd, 0xc39c, 0xf3ff, 0xe3de,
        0x2462, 0x3443, 0x0420, 0x1401, 0x64e6, 0x74c7, 0x44a4, 0x5485,
        0xa56a, 0xb54b, 0x8528, 0x9509, 0xe5ee, 0xf5cf, 0xc5ac, 0xd58d,
        0x3653, 0x2672, 0x1611, 0x0630, 0x76d7, 0x66f6, 0x5695, 0x46b4,
        0xb75b, 0xa77a, 0x9719, 0x8738, 0xf7df, 0xe7fe, 0xd79d, 0xc7bc,
        0x48c4, 0x58e5, 0x6886, 0x78a7, 0x0840, 0x1861, 0x2802, 0x3823,
        0xc9cc, 0xd9ed, 0xe98e, 0xf9af, 0x8948, 0x9969, 0xa90a, 0xb92b,
        0x5af5, 0x4ad4, 0x7ab7, 0x6a96, 0x1a71, 0x0a50, 0x3a33, 0x2a12,
        0xdbfd, 0xcbdc, 0xfbbf, 0xeb9e, 0x9b79, 0x8b58, 0xbb3b, 0xab1a,
        0x6ca6, 0x7c87, 0x4ce4, 0x5cc5, 0x2c22, 0x3c03, 0x0c60, 0x1c41,
        0xedae, 0xfd8f, 0xcdec, 0xddcd, 0xad2a, 0xbd0b, 0x8d68, 0x9d49,
        0x7e97, 0x6eb6, 0x5ed5, 0x4ef4, 0x3e13, 0x2e32, 0x1e51, 0x0e70,
        0xff9f, 0xefbe, 0xdfdd, 0xcffc, 0xbf1b, 0xaf3a, 0x9f59, 0x8f78,
        0x9188, 0x81a9, 0xb1ca, 0xa1eb, 0xd10c, 0xc12d, 0xf14e, 0xe16f,
        0x1080, 0x00a1, 0x30c2, 0x20e3, 0x5004, 0x4025, 0x7046, 0x6067,
        0x83b9, 0x9398, 0xa3fb, 0xb3da, 0xc33d, 0xd31c, 0xe37f, 0xf35e,
        0x02b1, 0x1290, 0x22f3, 0x32d2, 0x4235, 0x5214, 0x6277, 0x7256,
        0xb5ea, 0xa5cb, 0x95a8, 0x8589, 0xf56e, 0xe54f, 0xd52c, 0xc50d,
        0x34e2, 0x24c3, 0x14a0, 0x0481, 0x7466, 0x6447, 0x5424, 0x4405,
        0xa7db, 0xb7fa, 0x8799, 0x97b8, 0xe75f, 0xf77e, 0xc71d, 0xd73c,
        0x26d3, 0x36f2, 0x0691, 0x16b0, 0x6657, 0x7676, 0x4615, 0x5634,
        0xd94c, 0xc96d, 0xf90e, 0xe92f, 0x99c8, 0x89e9, 0xb98a, 0xa9ab,
        0x5844, 0x4865, 0x7806, 0x6827, 0x18c0, 0x08e1, 0x3882, 0x28a3,
        0xcb7d, 0xdb5c, 0xeb3f, 0xfb1e, 0x8bf9, 0x9bd8, 0xabbb, 0xbb9a,
        0x4a75, 0x5a54, 0x6a37, 0x7a16, 0x0af1, 0x1ad0, 0x2ab3, 0x3a92,
        0xfd2e, 0xed0f, 0xdd6c, 0xcd4d, 0xbdaa, 0xad8b, 0x9de8, 0x8dc9,
        0x7c26, 0x6c07, 0x5c64, 0x4c45, 0x3ca2, 0x2c83, 0x1ce0, 0x0cc1,
        0xef1f, 0xff3e, 0xcf5d, 0xdf7c, 0xaf9b, 0xbfba, 0x8fd9, 0x9ff8,
        0x6e17, 0x7e36, 0x4e55, 0x5e74, 0x2e93, 0x3eb2, 0x0ed1, 0x1ef0,
        ]
def crc16(data, crc=0):
    table = CRC16_XMODEM_TABLE
    for byte in data:
        crc = ((crc<<8)&0xff00) ^ table[((crc>>8)&0xff)^byte]
    return crc & 0xffff

This version is crc16 (CCIT). I use this in AX.25 APRS Ham radio stuff

#https://github.com/wb2osz/direwolf/blob/master/src/fcs_calc.c
CRC16_AX25 = [
   0x0000, 0x1189, 0x2312, 0x329b, 0x4624, 0x57ad, 0x6536, 0x74bf,
   0x8c48, 0x9dc1, 0xaf5a, 0xbed3, 0xca6c, 0xdbe5, 0xe97e, 0xf8f7,
   0x1081, 0x0108, 0x3393, 0x221a, 0x56a5, 0x472c, 0x75b7, 0x643e,
   0x9cc9, 0x8d40, 0xbfdb, 0xae52, 0xdaed, 0xcb64, 0xf9ff, 0xe876,
   0x2102, 0x308b, 0x0210, 0x1399, 0x6726, 0x76af, 0x4434, 0x55bd,
   0xad4a, 0xbcc3, 0x8e58, 0x9fd1, 0xeb6e, 0xfae7, 0xc87c, 0xd9f5,
   0x3183, 0x200a, 0x1291, 0x0318, 0x77a7, 0x662e, 0x54b5, 0x453c,
   0xbdcb, 0xac42, 0x9ed9, 0x8f50, 0xfbef, 0xea66, 0xd8fd, 0xc974,
   0x4204, 0x538d, 0x6116, 0x709f, 0x0420, 0x15a9, 0x2732, 0x36bb,
   0xce4c, 0xdfc5, 0xed5e, 0xfcd7, 0x8868, 0x99e1, 0xab7a, 0xbaf3,
   0x5285, 0x430c, 0x7197, 0x601e, 0x14a1, 0x0528, 0x37b3, 0x263a,
   0xdecd, 0xcf44, 0xfddf, 0xec56, 0x98e9, 0x8960, 0xbbfb, 0xaa72,
   0x6306, 0x728f, 0x4014, 0x519d, 0x2522, 0x34ab, 0x0630, 0x17b9,
   0xef4e, 0xfec7, 0xcc5c, 0xddd5, 0xa96a, 0xb8e3, 0x8a78, 0x9bf1,
   0x7387, 0x620e, 0x5095, 0x411c, 0x35a3, 0x242a, 0x16b1, 0x0738,
   0xffcf, 0xee46, 0xdcdd, 0xcd54, 0xb9eb, 0xa862, 0x9af9, 0x8b70,
   0x8408, 0x9581, 0xa71a, 0xb693, 0xc22c, 0xd3a5, 0xe13e, 0xf0b7,
   0x0840, 0x19c9, 0x2b52, 0x3adb, 0x4e64, 0x5fed, 0x6d76, 0x7cff,
   0x9489, 0x8500, 0xb79b, 0xa612, 0xd2ad, 0xc324, 0xf1bf, 0xe036,
   0x18c1, 0x0948, 0x3bd3, 0x2a5a, 0x5ee5, 0x4f6c, 0x7df7, 0x6c7e,
   0xa50a, 0xb483, 0x8618, 0x9791, 0xe32e, 0xf2a7, 0xc03c, 0xd1b5,
   0x2942, 0x38cb, 0x0a50, 0x1bd9, 0x6f66, 0x7eef, 0x4c74, 0x5dfd,
   0xb58b, 0xa402, 0x9699, 0x8710, 0xf3af, 0xe226, 0xd0bd, 0xc134,
   0x39c3, 0x284a, 0x1ad1, 0x0b58, 0x7fe7, 0x6e6e, 0x5cf5, 0x4d7c,
   0xc60c, 0xd785, 0xe51e, 0xf497, 0x8028, 0x91a1, 0xa33a, 0xb2b3,
   0x4a44, 0x5bcd, 0x6956, 0x78df, 0x0c60, 0x1de9, 0x2f72, 0x3efb,
   0xd68d, 0xc704, 0xf59f, 0xe416, 0x90a9, 0x8120, 0xb3bb, 0xa232,
   0x5ac5, 0x4b4c, 0x79d7, 0x685e, 0x1ce1, 0x0d68, 0x3ff3, 0x2e7a,
   0xe70e, 0xf687, 0xc41c, 0xd595, 0xa12a, 0xb0a3, 0x8238, 0x93b1,
   0x6b46, 0x7acf, 0x4854, 0x59dd, 0x2d62, 0x3ceb, 0x0e70, 0x1ff9,
   0xf78f, 0xe606, 0xd49d, 0xc514, 0xb1ab, 0xa022, 0x92b9, 0x8330,
   0x7bc7, 0x6a4e, 0x58d5, 0x495c, 0x3de3, 0x2c6a, 0x1ef1, 0x0f78
]
def crc16_ccit(data, crc=0xffff):
    table = CRC16_AX25
    for byte in data:
        crc = ((crc) >> 8) ^ table[((crc) ^ byte) & 0xff];
    return (crc ^ 0xffff) & 0xffff

[stephanelsmith]

@Tangerino seeing your question actually asked for C, I have that too for crc16 XMODEM.

// Include MicroPython API.
#include "py/runtime.h"

#include "py/objstr.h"
#include "py/objtuple.h"
#include <stdlib.h>

STATIC const uint16_t CRC16_XMODEM_TABLE[] = {
        0x0000, 0x1021, 0x2042, 0x3063, 0x4084, 0x50a5, 0x60c6, 0x70e7,
        0x8108, 0x9129, 0xa14a, 0xb16b, 0xc18c, 0xd1ad, 0xe1ce, 0xf1ef,
        0x1231, 0x0210, 0x3273, 0x2252, 0x52b5, 0x4294, 0x72f7, 0x62d6,
        0x9339, 0x8318, 0xb37b, 0xa35a, 0xd3bd, 0xc39c, 0xf3ff, 0xe3de,
        0x2462, 0x3443, 0x0420, 0x1401, 0x64e6, 0x74c7, 0x44a4, 0x5485,
        0xa56a, 0xb54b, 0x8528, 0x9509, 0xe5ee, 0xf5cf, 0xc5ac, 0xd58d,
        0x3653, 0x2672, 0x1611, 0x0630, 0x76d7, 0x66f6, 0x5695, 0x46b4,
        0xb75b, 0xa77a, 0x9719, 0x8738, 0xf7df, 0xe7fe, 0xd79d, 0xc7bc,
        0x48c4, 0x58e5, 0x6886, 0x78a7, 0x0840, 0x1861, 0x2802, 0x3823,
        0xc9cc, 0xd9ed, 0xe98e, 0xf9af, 0x8948, 0x9969, 0xa90a, 0xb92b,
        0x5af5, 0x4ad4, 0x7ab7, 0x6a96, 0x1a71, 0x0a50, 0x3a33, 0x2a12,
        0xdbfd, 0xcbdc, 0xfbbf, 0xeb9e, 0x9b79, 0x8b58, 0xbb3b, 0xab1a,
        0x6ca6, 0x7c87, 0x4ce4, 0x5cc5, 0x2c22, 0x3c03, 0x0c60, 0x1c41,
        0xedae, 0xfd8f, 0xcdec, 0xddcd, 0xad2a, 0xbd0b, 0x8d68, 0x9d49,
        0x7e97, 0x6eb6, 0x5ed5, 0x4ef4, 0x3e13, 0x2e32, 0x1e51, 0x0e70,
        0xff9f, 0xefbe, 0xdfdd, 0xcffc, 0xbf1b, 0xaf3a, 0x9f59, 0x8f78,
        0x9188, 0x81a9, 0xb1ca, 0xa1eb, 0xd10c, 0xc12d, 0xf14e, 0xe16f,
        0x1080, 0x00a1, 0x30c2, 0x20e3, 0x5004, 0x4025, 0x7046, 0x6067,
        0x83b9, 0x9398, 0xa3fb, 0xb3da, 0xc33d, 0xd31c, 0xe37f, 0xf35e,
        0x02b1, 0x1290, 0x22f3, 0x32d2, 0x4235, 0x5214, 0x6277, 0x7256,
        0xb5ea, 0xa5cb, 0x95a8, 0x8589, 0xf56e, 0xe54f, 0xd52c, 0xc50d,
        0x34e2, 0x24c3, 0x14a0, 0x0481, 0x7466, 0x6447, 0x5424, 0x4405,
        0xa7db, 0xb7fa, 0x8799, 0x97b8, 0xe75f, 0xf77e, 0xc71d, 0xd73c,
        0x26d3, 0x36f2, 0x0691, 0x16b0, 0x6657, 0x7676, 0x4615, 0x5634,
        0xd94c, 0xc96d, 0xf90e, 0xe92f, 0x99c8, 0x89e9, 0xb98a, 0xa9ab,
        0x5844, 0x4865, 0x7806, 0x6827, 0x18c0, 0x08e1, 0x3882, 0x28a3,
        0xcb7d, 0xdb5c, 0xeb3f, 0xfb1e, 0x8bf9, 0x9bd8, 0xabbb, 0xbb9a,
        0x4a75, 0x5a54, 0x6a37, 0x7a16, 0x0af1, 0x1ad0, 0x2ab3, 0x3a92,
        0xfd2e, 0xed0f, 0xdd6c, 0xcd4d, 0xbdaa, 0xad8b, 0x9de8, 0x8dc9,
        0x7c26, 0x6c07, 0x5c64, 0x4c45, 0x3ca2, 0x2c83, 0x1ce0, 0x0cc1,
        0xef1f, 0xff3e, 0xcf5d, 0xdf7c, 0xaf9b, 0xbfba, 0x8fd9, 0x9ff8,
        0x6e17, 0x7e36, 0x4e55, 0x5e74, 0x2e93, 0x3eb2, 0x0ed1, 0x1ef0,
};

// crc16
// INPUT
//      buff_in_obj: bytearray, bytes, memoryview
// OUTPUT
//      returns U16 crc
STATIC mp_obj_t crc16(mp_obj_t buff_in_obj) {
        mp_buffer_info_t bufinfo;
        mp_get_buffer(buff_in_obj, &bufinfo, MP_BUFFER_READ);
    uint8_t *buff_in = bufinfo.buf;
    uint16_t i;

    uint16_t crc = 0;
    for(i=0; i<bufinfo.len; i++){
        crc = ((crc<<8)&0xff00) ^ CRC16_XMODEM_TABLE[((crc>>8)&0x00ff)^buff_in[i]];
    }
    //printf("crc:%u\n", crc);
    // mp_printf(MP_PYTHON_PRINTER, "crc:%u\n", crc);
    //mp_obj_t mp_buff_out = mp_obj_new_bytearray(2, buff_out);
    //return mp_buff_out;
    return mp_obj_new_int(crc);
}
// Define a Python reference to the function above.
STATIC MP_DEFINE_CONST_FUN_OBJ_1(crc16_obj, crc16);


// Define all properties of the module.
// Table entries are key/value pairs of the attribute name (a string)
// and the MicroPython object reference.
// All identifiers and strings are written as MP_QSTR_xxx and will be
// optimized to word-sized integers by the build system (interned strings).
STATIC const mp_rom_map_elem_t example_module_globals_table[] = {
    { MP_ROM_QSTR(MP_QSTR___name__), MP_ROM_QSTR(MP_QSTR_ccrc) },
    { MP_ROM_QSTR(MP_QSTR_crc16), MP_ROM_PTR(&crc16_obj) },
};
STATIC MP_DEFINE_CONST_DICT(example_module_globals, example_module_globals_table);

// Define module object.
const mp_obj_module_t cmodule_ccrc = {
    .base = { &mp_type_module },
    .globals = (mp_obj_dict_t *)&example_module_globals,
};

// Register the module to make it available in Python.
MP_REGISTER_MODULE(MP_QSTR_ccrc, cmodule_ccrc);

[rkompass]

Nice.
Given that in my library the tables are calculated at class instantiation one could load a c-compiled mpy version like above for the calculator. It would use an array of table values as extra argument.
That would be faster than all the calculators present in the lib now probably, but retain the universality (right- left-shift, many and flexible CRC definitions). Will think about that.

[Tangerino]

After some MP bugs here and there, now I can use the official MP binary releases. It would awesome if the CRC modules be native so avoiding to compile myself my modules.
I think this is a feature used everywhere and brings a big value by speeding up standard calculations.
Thanks a lot for the help.

[JL1306]

Would be nice and valuable if CRC 16 and possibly CRC32 were integrated into the firmware as fast compiled c functions.
I use Modbus RTU a lot and have been struggled in the past with relatively slow python code.

[zcattacz]

I am surprised no one gave https://github.com/madler/crcany a try before ? I usually stick to this one when coding in C or mpy. It's tables covered all the three not so common CRCs I encountered so far.

[rkompass]

crcany is a suite of programs that generalize CRC calculations, and that generate C code to compute and combine CRCs efficiently. Any CRC can be computed given the set of parameters that describe it. Those parameters are provided in the form as used by Greg Cook's catalog of over one-hundred CRCs

It certainly is a good resource. Creating C code is one way to deal with the many variants of CRC computation.
I was more interested in the symmetries and other generalizing features of CRCs and found that having a fast routine for one shift direction allows for using it to compute CRC of the other shift as well. No need to do a bit-reverse of the input data (which some libraries out there do, but which slows computation down), just reversing of the table data of defining polynomials and of the CRC result is necessary (which are single fast computations).

So I devised my lib around that principle. Definitions of CRCs from such catalogs as the one mentioned are already there and may be extended if desired.
Later I found that you need only a fast routine for crc32 for doing crc16, crc8, crc7 (all with less bits) as fast as well.
And we have a fast crc32 routine already in MP, which uses a fixed 4-bit table (16 4-bit words, which correspond to 2 32bit words), which is only 40% slower than optimal (8-bit table).

So the way I would go would be to make that table accessible as an argument, for defining other CRCs and develop a lib around that. This way the modification of MP binaries would be quite minimal, still allowing to implement all CRCs up to 32 bits.

[zcattacz]

Depending on the byte size, the generated C code are mostly identical, the tables in the generated C code are different depending on the CRC scheme. My major inconvenience on mpy is how to store the tables memory efficiently, if not compiling them into the firmware.