ECU Diagnostics – part 12 : OSI 7 Layers for Caterham Diagnostics

Communications protocols are often called protocol stacks… they’re layers of different protocols… one layered on top of the other from lowest level (hardware/physical) to highest level (software application).

This is where the OSI 7 Layer model comes in… it standardises how we think about the layers and allows us to talk about where each protocol and hardware/software sits in the layer stack.

More info on the OSI 7 Layer Model can found here.

If we now load the OSI model with the layers for the MBE ECU protocol stack then the table looks like this:

PurpleMeanie / MBE ECU / OSI model
Layer PDU MBE ECU OSI Function
Host
layers
7 Application Data Application (Dashboard, Wheel Computer etc)
6 Presentation mbe.py (My Python helper class)
5 Session MBE ISOTP (MBE’s adaptions to ISOTP)
4 Transport Segment, Datagram ISOTP / ISO 15765-2 (pythong-can-isotp & can-isotp kernel module)
Media
layers
3 Network Packet
2 Data link Frame CAN bus ISO 11898 (SocketCAN)
1 Physical Symbol

From the table above…

MBE ECU OSI Function

Application

At the highest OSI level we have the application layer. This is what the user sees, turning all the lower level layers into something pretty to see.

mbe.py

This is my helper class. It takes the more complicated layers below and provides a simpler abstraction layer for the application to use.

MBE ISOTP

I’ve called this layer the MBE ISOTP layer. It adds a communication layer on top of the ISOTP packetization later. This MBE ISOTP layer provides “commands” for config and data extraction from the ECU. It does so “on top” of the ISOTP layer below, or another way to think of it is to say the MBE ISOTP packets sit “inside of” of the ISOTP packets.

ISOTP

This is the standard used to extend CAN bus packets from 8 bytes up to a maximum of 4095 bytes per packet and sits on top of the CAN bus protocol.

It also provides for an additional paradigm called flow control. In theory flow-control should allow CAN bus devices to negotiate the timing between CAN bus frames that each device can support.  

HOWEVER – The MBE/SBD system of Easimap and ECU DO NOT USE FLOW CONTROL. This caused us some concern to start with and meant we had to adjust some of the ISOTP libraries we were using, see here.

CAN Bus

We’ve talked about this layer a lot, the most relevant post about it can be found here.

ECU Diagnostics – part 11 : Logic Analyzer on the CAN Bus

To test out my theories and the code I’d been writing, I needed a simple test and a way to make sure the code (and I) was doing what I thought it should be.

The code part was easy, take the JSON files I’d created from the Easimap EC2 file and use it to ask for some simple data from the car.

As it turns out the “checking the code” part was also fairly straight forwards. I’ve talked about the Saleae Logic Analyzers in the past and how good they are. It also turns out that one of the software decoder modules they ship as standard is a CAN bus decoder. The idea is that you capture a analysis trace and get the analyser software to decode the trace as CAN bus.

Saleae also have instructions on how to set up a CAN bus capture. They say that in most cases you can connect the analyser to CAN-L and set the analyser input to be high voltage (3.3v) and away you go.

Saleae Logic Analyzer (red thing) connected to PiCAN2 on Raspberry Pi. Yellow wire connected to CAN-L, Black wire to Ground.

You’ll notice that there’s also a TFT LCD sat on top of the Pi, that’s for a later project.

Analyzer Setup

With the analyser input set to 3v3 and a 50MHz sample rate I could capture the 1MHz activity on the CAN bus. I tried some captures at slower sample rates but they weren’t so reliable – clearly the analyzer needs to significantly oversample the signal to get the timing right.

Just for completeness, here’s a picture of the analyser leads connected to the PiCAN2…

Saleae Analyzer connected to Raspberry Pi and PiCAN2

 

Here’s a closeup of the connections…

Analyzer connections to PiCAN2 closeup

Anyways, that setup is well within the capabilities of the Saleae and gave very reliable results.

Saleae Logic Analyzer Setup for CAN bus Capture on Channel 0

Test Code

Just to check my theories I wrote this simple app to pull just a few data samples from the car’s ECU. I started with RT_ENGINESPEED (Engine RPM) and got back the result I was expecting (i.e. 0 RPM, as the car wasn’t running).

Here’s the code, which can also be found here.

# TestECU
# Application for testing python-isotp-mbe class
#
# 2019-08-26 John Martin
#

import logging
import argparse
import json
import pprint
import mbe
import curses

version = "0.1"

if (mbe.test_mode):
  variables_to_follow = [
      'RT_ENGINESPEED',
    'RT_AIRTEMP1(LIM)',
    'RT_COOLANTTEMP1(LIM)',
    'RT_BATTERYVOLTAGE(LIM)',
    'RT_SOFTCUTTIME',
    'RT_HARDCUTTIME'
  ]
else:
  variables_to_follow = [
    'RT_ENGINESPEED'
  ]

def main():
  parser = argparse.ArgumentParser(prog='mbepcap2txt', description='Takes an pcap with ISOTP formatted MBE transactions and makes it human readable.')
  parser.add_argument('--interface',     '-i',                   help='The can interface to open', required=True)
  parser.add_argument('--variables',     '-v',                   help='Input MBE variables filename', required=True)
  parser.add_argument('--query_id',      '-q',                   help='CAN query ID (default 0x0cbe1101)', default=0x0cbe1101)
  parser.add_argument('--response_id',   '-r',                   help='CAN resdponse ID (default 0x0cbe0111', default=0x0cbe0111)
  parser.add_argument('--loglevel',      '-l',                   help='Logging level to show', choices=['INFO','DEBUG','WARNING', 'ERROR', 'NONE'], default="ERROR")
  parser.add_argument('--logfile',       '-f',                   help='If set logging will be sent to this file')
  parser.add_argument('--version',       '-V', action='version', version='%(prog)s '+version)

  args = parser.parse_args()

  logging_level = getattr(logging, args.loglevel, None)
  logging.basicConfig(level=logging_level, filename=args.logfile, filemode='w')

  ecu = mbe.mbe()
  
  ret = ecu.set_options(args.variables, args.query_id, args.response_id, args.interface)
  if(not ret):
    logging.error("Unable to set options")
    exit()

  if(ecu.add_variable_list_to_follow(variables_to_follow) != len(variables_to_follow)):
    logging.warning("Ooops, didn't add all the vars I wanted to")
  else:
    logging.info("Added all the variables we expected")

  ecu.bind()

  results = dict()
  if (ecu.process_all_pages(results) != False):
    logging.debug(pprint.pformat(results))

if __name__ == '__main__':
  main()

When we run the code above it makes a single request to the car for Engine RPM and displays the results. Here are the ISOTP CAN bus packets for this test:

REQUEST:  0100000000f87c7d
RESPONSE: 810000

As you can see in first packet, there’s a data request (0x01)  to page 0xf8 with 0x7c and 0x7d as the requested data.

Then in the response packet there’s a data response (0x81) with 0x0000 as the RPM.

Here’s the 3 raw CAN bus frames that correspond to those ISOTP packets:

#     CAN ID     Data
[290] 0x0cbe1101 10 08 01 00 00 00 00 f8
[291] 0x0cbe1101 21 7c 7d
[292] 0x0cbe0111 03 81 00 00 00 00 f8 7c

And here’s what that looked like in the logic analyser. The three images are for each of the three CAN bus frames (you might need to click on the images to see them full scale)…

CAN bus request: 0x0cbe1101 10 08 01 00 00 00 00 f8
CAN bus request: 0x0cbe1101 21 7c 7d
CAN bus response: 0x0cbe0111 03 81 00 00 00 00 f8 7c

You can see details of the CAN bus protocol in full technicolor. The analyser software doesn’t show the start but you can see the ID, data and CRC. A more detailed discussion on the CAN bus protocol can be found in this post.

So my software is able to send a good ISOTP CAN bus request to the car and we get a good response back.

Lets look at all we know about the various protocols now.

ECU Diagnostics – part 10 : Decoding EC2 Files

Now need to start proving our theories about how Easimap is actually talking to the ECU. And it seemed to me that if I was going to prove anything then I’d need to pass a lot of data through any theory I came up with, and that meant writing a program – since bit bashing hundreds of packets of CAN bus data was probably going to take as long to decode by hand as it would to write a program to do it.

And if I was going to write a program then it was going to need to translate the binary data on the CAN bus into something human readable. Of course that’s what Easimap uses the EC2 files for and I wanted to use that same information in any programs I wrote.

The EC2 files are plain text files using a similar, but not quite the same, layout as legacy Windows ini files. There a lots of programs that now use this file format and it seems MBE/SBD have come up with their own variation – at least I’ve not seen anything quite like it before.

It uses the regular square brackets “[ ]” to create sections in the file, but it then nests these sections to create hierarchies. And it means that standard libraries to read in an EC2 file won’t work. Time to write some code then…

EC2 File Sections

The 9A4 EC2 file has the following top-level sections, each contained in the EC2 file and having square brackets around them:

  • HISTORY
  • PROPERTIES
  • PARAMETER PROTOTYPES
  • NUMERIC SCALES
  • STRING SCALES
  • PARAMETER DEFINITIONS
  • COLLECTIONS
  • ALARMS
  • SPECIAL FUNCTIONS
  • MAPPING VECTORS
  • MAPPING CONTROLS
  • SETUP PROTOTYPES
  • SETUP DEFINITIONS
  • MAP PROGRAMS
  • VARIANTS
  • SPECIAL INTERFACE PROTOTYPES
  • MATRICES

So, in the EC2 file the History section has a line with [ HISTORY ] in it.

There’s clearly a lot of those sections that will be interesting to look into at some point. But I had a very focused desire to get the raw sensor diagnostic data out of the car and that meant I only needed to look at three sections for the moment.

Parameter Definitions

After a bunch of trial and error it seemed I needed to use the Parameter Definitions to get a list of all the data variables that the car could divulge.

So for Engine RPM the corresponding Parameter Definition is:

[RT_ENGINESPEED]
Number of Dimensions = 0
Page = F8
Address = 237C
Bytes per Cell = 2
0 = SCALE_ENGINESPEED

That allowed me to figure out what the request was that I needed to make to the ECU in order for the ECU to respond with Enging RPM. See the previous post for a breakdown of that.

It also told me that I was going to get two bytes of data back from the ECU (Bytes per Cell = 2).

But 2 bytes of data can range from 0 to 65535 (assuming its a positive integer). And I wasn’t sure how to interpret the 2 bytes. Other data variables, like battery voltage, were going to be floating point numbers and while Engine RPM could be a 2 byte integer with no scaling or offset, that wasn’t going to be the case for most of the other data coming from the car.

Numeric Scales

That’s where the Numeric Scales top level section comes in…

It turns out that the line:

0 = SCALE_ENGINESPEED

refers to a SCALE_ENGINSPEED section in the Numeric Scales section of the EC2 file. The “0 = ” bit refers to the fact that this is the zeroth dimension for the variable, something that becomes important for 1D, 2D and 3D maps where there are more scales defined for each of the axies on the map. The definition of SCALE_ENGINESPEED looks like this:

[SCALE_ENGINESPEED]
Units = RPM
Scale Minimum = 0.000000
Scale Maximum = 65535.000000
Display Minimum = 0.000000
Display Maximum = 12000.000000
Display Interval = 1000.000000

I suspect that’s fairly obvious what’s going on here, but the idea is that these Numeric Scales set the limits for the data coming back from the car, along with any scale factor that needs to be applied and any offset.

For instance, the temperate variables have a scale (allowing them to be floating point numbers when the result is divided by the scale) and offsets (meaning that the data coming from the car needs to be offset by a particular value, allowing the number to also be negative – as might happen with air temperature on a cold day).

You can also see that the Numeric Scale also sets the characters we can use for the Units, RPM in this case.

Finally, I wanted to be able to see the data coming from the car in a human readable form. And while RT_ENGINESPEED is reasonably simple to understand, there are other variables that are much more obtuse, like RT_TPSFUEL+TRIMBANK1. 

Parameter Prototypes

That’s where the Parameter Prototype section comes in with English definitions of what each variable is. For instance, RT_ENGINESPEED has this parameter prototype:

RT_ENGINESPEED, 
Engine Speed,
EngSpeed,
Engine Speed in rpm as measured by the crankshaft sensor. This is
the primary input for the fuel and ignition maps

I spread that definition across a few lines, it’s on a single line in the EC2 file.

One thing to note about these Parameter Prototypes is that at the end of the line in the EC2 file, you can get the word “DISABLED”. I ignored this at first but soon came to realise that it really did mean “disabled”. If I didn’t take account of it in the code I wrote, then I found there were variables defined more than once. Only one of the definitions was not disabled, so by ignoring the disabled ones I was able to get a clean list of valid variables.

ec2parse.py

Now I knew the layout of the EC2 file, I could write some code. My weapon of choice here is Python. I’m reasonably new to Python but there’s a huge library base now behind Python and I can code with it on Linux, Mac and PC reasonably easily. 

The code for ec2parse.py can be found here. The code is far from pretty… but it does a job. I may go and tidy it up when I get a chance (but I think everyone says that and I always advocate writing neat code in the first place so you don’t have to go back to it later – do as I say, not a I do!).

This Python code will take an EC2 file and create an output file that can be used later, i.e. in the mbepcap2txt.py example below. For the moment I’ve been using the JSON output that I created which creates a formatted file that another Python file can read (use) instead of it having to decode the EC2 file every time.

mbepcap2txt.py

Once I’d written this parsing code for the EC2 files I also the wrote a test app to take a CAN bus Wireshark capture file (.pcap or .pcapng) and turn it into something human readable – remember this is CAN bus being turned into a SocketCAN format, decoded as CAN then as ISOTP – phew!.

This allowed me to take many hundreds of messages sent between Easimap and the ECU and verify that I had all my theories validated before I actually used any of this software I’d written to send any messages to the car…. I didn’t want to have to fork out for a new ECU, or even worse a new engine because I’d sent some random message to the ECU.

And it all worked… here’s some output in something more human readable that the CAN bus messages:

REQUEST: 0100000000f83031363744454c4d4e4f50515a5b5c5d646a6b7c7d9e9fa0a1d8d9dadb
RESPONSE: 814e39094a9c52b886e8844e39a80d70a000b04f00005695800900800080
RT_THROTTLEANGLE1(RAW)=1.1192 V ( Throttle Angle 1 (Raw) )
RT_AIRTEMP1(LIM)=16.273 ( Air Temp )
RT_COOLANTTEMP1(LIM)=21.632 ( Coolant Temp )
RT_COOLANTFUELFACTOR=110.5 ( Coolant Fuel Factor )
RT_AIRTEMPFUELFACTOR=3.8346 ( Air Temp Fuel Factor )
RT_THROTTLEANGLEINCREASING=1.1192 V ( Throttle Angle Increasing )
RT_TPSFUEL+TRIMBANK1=5.6013 ms ( TPS Fuel + Trim )
RT_TPSVSSPEEDIGN+TRIM1=4.7938 ( TPS vs Speed Ign + Trim )
RT_THROTTLESITE1=0.0 Site ( Throttle Site 1 )
RT_BAROSCALEDLIM=1.04 Bar ( Baro Pressure )
RT_ENGINESPEED=0.0 RPM ( Engine Speed )
RT_BATTERYVOLTAGE(LIM)=11.667 V ( Battery Voltage )
RT_BATTERYVOLTAGECOMP=0.48725 ms ( Battery Voltage Comp )
RT_MAPPINGPOT1LIM=0.0015259 ( Mapping Pot 1 )
RT_MAPPINGPOT2LIM=0.0015259 ( Mapping Pot 2 )

#[28]# ISOTP REQUEST :0100000000f9babbbcbd
#[29]# ISOTP RESPONSE :81781edc1e
RT_SOFTCUTTIME=7800.0 RPM ( Soft Cut )
RT_HARDCUTTIME=7900.0 RPM ( Hard Cut )

But before I could do some real damage to the car, there was one more test I wanted to do to make sure everything was going to be as safe as I could make it. I wanted to connect up a Logic Analyser to the CAN bus so I could make sure the communications were working as expected…

 

 

 

ECU Diagnostics – part 9 : The Easimap Protocol Theory

Ok. so we’ve set ourselves up to be able to look at what is going on with a Caterham diagnostics port, we’ve talked a bit about the theory of what we might see from the port and we’ve also seen that Easimap talks to the ECU using a higher level protocol called ISO 15765-2 (ISOTP). We also saw how Easimap is asking the ECU for its model and serial number.

Easimap Data Request Protocol

At about this time, we also made a bit of a leap in our understanding of the communications between Easimap and the car.

We’d seen how Easimap sending 0x04000d to the car gave a response of the model and serial numbers (0xe4000d23394134626535333000). Notice how the first byte of the request is 0x04 and the first byte of the response is 0xe4. Seemed to me as though the response was echoing the 4 and setting some higher order bits creating a response code of 0xe4… Hmm…

If we look further down the communications that Easimap sends to the car (with the default gauges and pages set in Easimap) then we can also see some patterns.

It seemed to me that the first byte of the model/serial request was going to be a message type (0x04 means get some config data).

Looking through the next communications we see Easimap send:

0x0100000000126667a8a9

and gets a response of:

0x81aaaa1600

If we think back to the 0x04 and 0xe4, request/response, of the model/serial exchange then this new exchange looks like a request/resonse of 0x01/0x81. That’s the same sort of pattern… a request of 0x01 and a response with the topmost bit set. That’s starting to make sense, but what of all the other data?

It was at about this time that James (Aerobod) had been looking at his car and had seen a pattern where there was an Lowest Significant Byte (LSB) and Most Significant Byte (MSB) being sent periodically that matched some of the data he was looking for.

It then dawned on me that my previous hunch about pages and addresses in the EC2 files was starting to make sense. If we look at our request to the car of:

0x0100000000126667a8a9

then the first byte we know about (0x01) and then there are some more bytes of 0x00 with a 0x12 next, that could a page. Then there are sequences of similar numbers 0x66 followed by 0x67, then 0xa8 followed by 0xa9. They could be addresses.

I searched the EC2 file for 0x6667 and 0xa8a9 but there wasn’t anything that made sense.

It then occurred to me that perhaps the pages and addresses in the EC2 file weren’t being used exactly as written in the file. And if James’ thinking about LSB and MSB was right then perhaps the 0x6667 was the LSB of the address in the EC2 file followed by the same LSB+1.

With the data we have above that would mean we needed to look in the EC2 file for a variable with a page of 0x12 and address LSB of 0x66. A quick search in our EC2 file turns up:

[RT_IGNITIONADVANCEBANK1]
Number of Dimensions = 0
Page = 12
Address = 6766
Bytes per Cell = 2
0 = SCALE_NADV

Some of the right data was present, the 12, 66 and 67 were there. And it seemed to indicate that this was related to Ignition Advance Bank1. Ok, that could be, our Caterhams could have something like that in them.

Next was to look at the response from the car:

0x81aaaa1600

We know that 0x81 is the response type for a request of 0x01. That leaves data of 0xaaaa1600. That’s probably 0xaaaa and 0x1600. Neither seemed to make much sense for an Ignition Advance Bank 1 response. It could be right, it could be wrong.

At this point I also started to pull apart the test.lsn file I’d found in the Easimap configuration directories on the PC. That file seemed to be the setup Easimap was using to say what gauges and displays to put on the screen for each “page” of the software display. It had a section like this:

[Parameters]
RT_THROTTLESITE1, Throttle Position|Throttle Position Limits and Default Setup, Throttle Site 1
RT_BATTERYVOLTAGECOMP, Fuel|Voltage Fuel Compensation, Battery Voltage Comp
RT_ENGINESPEED, Engine Speed Setup & Limiter, Engine Speed
RT_IGNITIONADVANCEBANK1, Ignition, Ignition Advance
RT_TPSVSSPEEDIGN+TRIM1, Ignition, TPS vs Speed Ign + Trim
RT_INJECTIONTIMEA, Fuel, Final Injection Time
RT_COOLANTTEMP1(LIM), Sensors|Coolant Temperature, Coolant Temp
RT_AIRTEMP1(LIM), Sensors|Air Temperature, Air Temp
RT_MAPPINGPOT1LIM, Mapping Box|Mapping Pot 1, Mapping Pot 1
RT_MAPPINGPOT2LIM, Mapping Box|Mapping Pot 2, Mapping Pot 2
RT_COOLANTFUELFACTOR, Fuel|Coolant Temperature Compensation, Coolant Fuel Factor
RT_BATTERYVOLTAGE(LIM), Sensors|Battery Voltage, Battery Voltage
RT_AIRTEMPFUELFACTOR, Fuel|Air Temperature Compensation, Air Temp Fuel Factor
RT_DUTYCYCLEA, Fuel|Duty Cycle, Duty Cycle
RT_TPSFUEL+TRIMBANK1, Fuel, TPS Fuel + Trim
RT_SOFTCUTTIME, Engine Speed Setup & Limiter, Soft Cut
RT_HARDCUTTIME, Engine Speed Setup & Limiter, Hard Cut
RT_THROTTLEANGLE1(RAW), Throttle Position|Throttle Position Limits and Default Setup, Throttle Angle 1 (Raw)
RT_ENGINERUNTIME, Data Logging, ERT
RT_ECUSTATUS, Fault Conditions, ECU Status
RT_BAROSCALEDLIM, Sensors|Barometric Pressure, Baro Pressure
RT_THROTTLEANGLEINCREASING, Throttle Position|Throttle Position Limits and Default Setup, Throttle Angle Increasing
RT_BAROFUELCOMP, Fuel|Barometric Pressure Fuel Compensation, Barometric Pressure Fuel Compensation
RT_CRANKCOUNT, Development, Crank Count

That was clearly a list of variables (in upper case) that Easimap was going to request data for from the car.

And an obvious candidate for a search in there was RT_ENGINESPEED, or Engine RPMs. I knew that I’d not had the car engine running when I did my packet captures, so there was a good chance that if I found the request for RPM then I could see a response of with zeros back from the car.

Looking in the EC2 file, you can find definition of RPM variable as:

[RT_ENGINESPEED]
Number of Dimensions = 0
Page = F8
Address = 237C
Bytes per Cell = 2
0 = SCALE_ENGINESPEED

Ok. So, we’ll looking for a request with a page of 0xf8 and an address that’s either 0x237c or possibly 0x7c7d.

Looking through the CAN bus as ISOTP output again we find this request:

0100000000f83031363744454c4d4e4f50515a5b5c5d646a6b7c7d9e9fa0a1d8d9dadb

and this response:

814e39094a9c52b886e8844e39a80d70a000b04f00005695800900800080

In the request there’s the 0x01 at that start that is some kind of data request and the corresponding 0x81 in the response. We then see 0xf8 after the 4 x 0x00’s. Ok, that’s looking good, we’re looking for a page of 0xf8 for RT_ENGINESPEED.

We can also see that 3/4 of the way along the request we can see 0x7c7d. If we break the request down into the bit at the start (pre-amble) of 0x0100000000f8 and then the pairs that could be LSB/MSB combinations, then we get this:

0100000000f8
3031
3637
4445
4c4d
4e4f
5051
5a5b
5c5d
64
6a6b
7c7d
9e9f
a0a1
d8d9
dadb

That’s all looking good for the LSB/MSB theory of LSB/LSB+1 and shows us that 0x7c7d is 19 bytes into the data section. I also took note that some data being requested has only one byte of request information… I’ll need to remember that later.

Now if we look at the response from the car and break it down like the request data and slot in the request data on the left:

 Request         Response
0100000000f8 = 81
3031 = 4e39
3637 = 094a
4445 = 9c52
4c4d = b886
4e4f = e884
5051 = 4e39
5a5b = a80d
5c5d = 70a0
64 = 00
6a6b = b04f
[7c7d = 0000]
9e9f = 5695
a0a1 = 8009
d8d9 = 0080
dadb = 0080

Then… low and behold… where I’ve put brackets around the line, we can see that our theory about RT_ENGINESPEED needing a request of 0x7c7d is returning 0x0000 in exactly the right spot in the response… showing that the car isn’t running.

That’s good enough for me… I think my theory is as good as I’m going to get, I need to write some code to test this out.

ECU Diagnostics – part 8 : Easimap uses ISOTP (sort of)

Ok, that’s enough of the theory about mapping. Lets get back to what Easimap is doing when it talks to the car.

When we left this subject last we had got to the point where we could see what Easimap was sending and receiving on the CAN bus, but we didn’t know what it meant. We had seen a set of startup packets like this:

#  Time        ID         Data
28 0.265548270 0x0cbe1101 03 04 00 0d 26 40 42 43
29 0.266543253 0x0cbe0111 10 0d e4 00 0d 23 39 41
30 0.266872637 0x0cbe0111 21 34 62 65 35 33 30 00
37 0.326052544 0x0cbe1101 03 04 00 5e 26 40 42 43
38 0.327045620 0x0cbe0111 07 e4 00 5e 2b 07 01 00

Then we get this set of 31 packets that repeat continually but with small changes (probably the “real” data from the ECU):

#   Time        ID         Data
80 0.736160089 0x0cbe1101 10 0a 01 00 00 00 00 12
81 0.736958836 0x0cbe1101 21 66 67 a8 a9 00 00 12
82 0.738230722 0x0cbe0111 05 81 aa aa 16 00 12 66
84 0.749027454 0x0cbe1101 10 09 01 00 00 00 00 1a
85 0.749733313 0x0cbe1101 21 52 5c 5d 00 00 00 1a
86 0.750875498 0x0cbe0111 04 81 84 00 00 00 1a 52
89 0.767924313 0x0cbe1101 10 0a 01 00 00 00 00 e2
90 0.768777632 0x0cbe1101 21 cc cd ce cf 00 00 e2
91 0.769825875 0x0cbe0111 05 81 ff ff ff 07 e2 cc
93 0.780880102 0x0cbe1101 10 23 01 00 00 00 00 f8
95 0.781565869 0x0cbe1101 21 30 31 36 37 44 45 4c
96 0.782301116 0x0cbe1101 22 4d 4e 4f 50 51 5a 5b
97 0.783061863 0x0cbe1101 23 5c 5d 64 6a 6b 7c 7d
98 0.783818702 0x0cbe1101 24 9e 9f a0 a1 d8 d9 da
99 0.784526524 0x0cbe1101 25 db 9f a0 a1 d8 d9 da
100 0.785768596 0x0cbe0111 10 1e 81 5e 39 cc 48 dc
101 0.786095776 0x0cbe0111 21 50 9c 89 1e 85 5e 39
102 0.786437863 0x0cbe0111 22 a8 0d 66 a0 00 b0 4f
103 0.786777913 0x0cbe0111 23 00 00 56 95 80 09 00
104 0.787102630 0x0cbe0111 24 80 00 80 a1 d8 d9 da
106 0.798260523 0x0cbe1101 10 0a 01 00 00 00 00 f9
107 0.799080454 0x0cbe1101 21 ba bb bc bd 00 00 f9
108 0.800070197 0x0cbe0111 05 81 78 1e dc 1e f9 ba
110 0.810033090 0x0cbe1101 10 09 01 00 00 00 00 fa
111 0.810853632 0x0cbe1101 21 64 65 6c 00 00 00 fa
113 0.812042780 0x0cbe0111 04 81 8c 1e 00 00 fa 64
115 0.821498922 0x0cbe1101 10 0e 01 00 00 00 00 fd
116 0.822377593 0x0cbe1101 21 20 24 25 26 40 42 43
117 0.823099192 0x0cbe1101 22 4d 24 25 26 40 42 43
118 0.824343987 0x0cbe0111 10 09 81 40 00 00 01 80
119 0.824665352 0x0cbe0111 21 b0 40 40 26 40 42 43

We’ve also seen that I’d had a diversion looking for bit-shifted pages and addresses which got us nowhere.

Lets get back to Wireshark again…

Wireshark Sub Dissectors

I’d played around with a lot of the CAN bus packet captures we’d taken from Easimap talking to our ECUs and I’d taken a bit of a diversion to fix Wireshark’s OBD-II dissector. I’d also spent time trying to see if Wireshark could dig deeper into the packets with it’s sub-dissectors.

Wireshark works by iterative dissection – meaning that it decodes a packet then passes that packet on to further dissectors, called sub-dissectors, to see if they can further decode the packet. So, Wireshark’s CAN dissector defines further sub-dissectors for these protocols:

  • OBD-II (which I’ve talked about before)
  • ISO 15765
  • CanOpen
  • DeviceNet
  • J1939
  • ISOBUS
  • AUTOSAR NM

I thought I had tried to decode our Easimap stream with all of them before, but after coming up blank with my correlator and having read the Car Hacker’s Handbook, I thought I’d have another go. The Car Hacker’s Handbook talks about car manufacturers using proprietary protocols on their CAN bus implementations but it also talks about progress in standardising some of the protocols used.

And… low and behold… when I selected the ISO 15765 sub-dissector to decode our Easimap communications, it was showing some promise. Once I’d issolated just the Easimap traffic then it looked something like this:

[ 2] ISOTP OTHER_REQUEST :04000d
[ 4] ISOTP OTHER_RESPONSE :e4000d23394134626535333000
[ 5] ISOTP OTHER_REQUEST :04005e
[ 6] ISOTP OTHER_RESPONSE :e4005e2b070100

[ 8] ISOTP REQUEST :0100000000126667a8a9
[ 9] ISOTP RESPONSE :81aaaa1600
[11] ISOTP REQUEST :01000000001a525c5d
[12] ISOTP RESPONSE :81840000
[14] ISOTP REQUEST :0100000000e2cccdcecf
[15] ISOTP RESPONSE :81ffffff07
[21] ISOTP REQUEST :0100000000f83031363744454c4d4e4f50515a5b5c5d646a6b7c7d9e9fa0a1d8d9dadb
[26] ISOTP RESPONSE :814e39094a9c52b886e8844e39a80d70a000b04f00005695800900800080
[28] ISOTP REQUEST :0100000000f9babbbcbd
[29] ISOTP RESPONSE :81781edc1e
[31] ISOTP REQUEST :0100000000fa64656c
[32] ISOTP RESPONSE :81ef1d00
[35] ISOTP REQUEST :0100000000fd202425264042434d
[37] ISOTP RESPONSE :814000000180b04040

ISOTP Basics

Ok. Lets step back a bit here. More info on ISOTP (ISO 15765-2) can be found here. It defines a bunch of packet types that can be used to extend the regular CANbus communications from packets of 8 up to packets with a total of 4095 bytes. To do that it steals the first byte or two (depending on the message type) to say what ISOTP packet type this is, and to tell us how long (in bytes) the packet is.

Those first two bytes are described as follows:

Table of ISOTP Message Types and Bit Patterns

For those wanting some help with what that means:

  • The top 4 bits (7..4) of byte zero define the message type as follows:
    • 0 = “Single” Byte Message
    • 1 = “First” byte of a multi-byte message
    • 2 = a “Consecutive” byte sent after a “First” byte
    • 3 = “Flow” Control message
  • The bottom 4 bits of byte zero change their meaning depending on which message type is being sent
  • If data is being sent (anything other than a flow control message) then the data starts in either byte 1 or byte 2 and continues for “size” bytes

Lets look at an example of what that means…

ISOTP Example Transmission

From the CAN bus packets shown at the top of this post, we have the first packet viewed as a CAN bus packet as follows:

#  Time        ID         Data
28 0.265548270 0x0cbe1101 03 04 00 0d 26 40 42 43

So the data part of that is:

03 04 00 0d 26 40 42 43

If we look at the top 4 bits (0x0) of the first byte (0x03), we can see that we have a message type of ‘0’. And from the table above message type ‘0’ means we are sending one CAN bus frame and the bottom 4 bits of byte 0 tell us how many data bytes we have in this frame, which is 3.

So, even though there were 8 bytes sent on the CAN bus, only the first byte (0x03) and the subsequent 3 bytes (0x04, 0x00, 0x0d) are of any use. The last 4 bytes (0x26, 0x40, 0x42, 0x43) are junk. In some implementations of ISOTP the devices on the CAN bus would insert “padding” where these junk bytes are here, and we might see them as 0x00 or 0xff or 0xaa, or some other constant but essentially useless data.

In theory Easimap and the ECU don’t need to send these junk or padding bytes. And in my experiments the ECU’s are happy to be sent them or not.

That’s decoding the CAN bus packet and interpreting it as ISOTP.

I also wrote some Python code to decode and display ISOTP packets and it shows us the output of this single CAN bus frame as this:

[ 2] ISOTP OTHER_REQUEST :04000d

Which matches up with our manual decoding of that packet, the data is 0x04, 0x00, 0x0d.

But There’s a Twist – No Flow-Control

As you can see from the “Table of ISTOP Messages and Bit Patterns” above, there’s a flow-control message type defined… but I’ve not shown any flow-control messages so far.

ISOTP is a simple protocol, buts also not just a few lines of code to write the software for. I’d anticipated using a Python compatible library to decode this layer and didn’t really want to write my own.

However, flow-control is an implicit part of the ISOTP protocol and any the libraries I looked at had it intrinsic to the code.

I looked at two ways of doing ISOTP. Both were using the Python-CAN-ISOTP library. And this can either implement ISOTP in user space (meaning it is written in Python, communicates with SocketCAN directly, and is not a kernel module) or it can sit as a thin “wrapper” for the can-isotp kernel module.

I looked at hacking the user space Python version of python-can-isotp but decided that was going to be a lot of work, and there were speed and software architecture benefits to using the can-isotp kernel module.

So, I got to work “adjusting” (some may say hacking) the can-isotp module from Oliver Hartkopp. I changed the code to not wait for flow-control messages when receiving packets (this is almost exactly the same as how can-isotp does the same when in listening mode) and stopped it from sending flow-control when sending packets. All in all it was only some 40 changes to Oliver’s original code.

Normally I’d put a link to Oliver’s GitHub repository here, but that might confuse people.  What I’m going to do is link here to my modified version of his repo. That way you’ll not go to all the trouble of installing his code, only to find it won’t work on an MBE ECU. If you want to find his unadulterated code then it can be found from my version of his repo. It might be that Oliver can support this removal of flow-control, but I can fully see how he might not want to do that – you’re supposed to use flow-control with ISOTP and so why would a library supporting the standard include some perversion of that standard. We’ll see how this one plays out and I might update this post to account for where it goes.

But also, for the record… Oliver has done an awesome job with his kernel module. I almost feel embarrassed to have chopped out the flow-control segment, which is a big part of the code. Sorry Oliver!

Back to our ECU’s: We’re not sure why MBE don’t implement flow-control. It’s possibly because it was thought to be too complicated. Or perhaps the ECU’s can respond fast enough not to need the timing / throttling that flow-control offers. Or perhaps it’s both.

One thing we’re going to have to keep in mind though… it might be the case that flow-control is required in an MBE ECU application. It might be that Easimap is doing the backing-off of the transmission to the ECU and if we “hit” the ECU to hard then it might not be able to keep up. We’ll have to keep an eye on this as we develop our applications going forwards. We certainly don’t want to overload the ECU and for it not to be able to do what it’s really meant to… i.e. keeping the engine running properly… that would not go well if we screw that up!

But with flow-control removed from the ISOTP layer we’re off to the races.

It All Starts to Make Sense

As I looked through the communications between Easimap and the car, it was clear that it made a lot of sense when it was decoded as ISOTP. I had no idea what 0x04,0x00,0x0d meant at this time, but at least I had a sense that another layer of the onion was understood.

This made even more sense when I looked at the response from the car…

Easimap was sending 0x04000d and the car was responding with 0xe4000d23394134626535333000 (I’ve just joined all the bytes together here instead of adding the 0x to each one).

That response may seem a bit random. But if we look at it by converting it to ASCII then we get the following:

...#9A4be530.

That also looks a bit random, but if we ignore the dots (which mean the byte is an unprintable ASCII character) then we’re left if 9A4be530… and pulling that apart… clearly 9A4 is the model of the ECU… and by checking back with Easimap… be530 is hexadecimal (0xbe530) and is the serial number of my ECU.

Voila!

By decoding the CAN bus data as ISOTP we’d managed to extract a string (of characters) that was the model of the ECU and the ECU’s serial number. That meant we had some meaningful data and we’re on the right track now.

It also made even more sense when we look at what Easimap does when connected to the car. When Easimap first communicates with the car, it pops up a window in the software showing that it needs to load a particular EC2 file. It must do that because it has found something out about the ECU it’s talking to. It makes perfect sense that as the first communications with the car, it will ask for its model number and serial number so it can go and load the correct EC2 file. It all adds up.

We just need to figure out what all the rest of the communications between Easimap and the car are.

And to do that, it seemed that I needed to know more about what the Ec2 file was telling Easimap when it loads at the start of the communications. So next up we’ll dive back into the EC2 files again…