ECU Diagnostics – part 7 : ECU Maps and Mapping

People often talk about “mapping” a car. When they do that they’re usually interested in one or more of three things:

  • To increase its performance
  • Improve its fuel economy
  • Improve its drivability

So what is mapping?

In technical terms a map is a translation, or conversion, function that takes an input and produces an output. We talk about “mapping” the input to the output. i.e. we take lots of different input data and convert it into another form as an output. This can be done by way of a formula, also known as a function, or with a lookup table, which we’ll talk about later.

Mapping happens in many computing situations, power stations, chemical plants, washing machines, in car ECUs, almost anywhere there is a computer handling physical process.

Lookup Tables

Lookup tables are typically how ECU’s convert the input sensor data into the output data that gets fed into controlling the engine. They are like multiplication times tables we used at school, i.e. they’re a simple way of taking a question (what’s 4 x 5?) and “looking up” the answer (20). In this example 4 is one input, 5 is another input and 20 is the output. And that gets us a two dimensional lookup table.

For times-tables, we “look up” the result by searching across the row and down the column of the table to find the result of the multiplication. This is what ECU’s use the maps for… they are lookup tables.

1 to 8 Times Tables: 4 x 5 = 20

The times table above is shown flat on the page, but we can also draw it in three dimensions, like this:

1 to 8 Times Table drawn in three dimensions

That 3D diagram is the exact same times-table as was drawn before, but now its in 3D rather than 2D.

One type of engine map is exactly like these times tables. The rows on one axis of the map are one input (i.e. RPM) and along the other axis are the columns with the other input (i.e. throttle position). So, this particular example “map” might take RPM on one axis and throttle position on the other and the number we might get when we do the lookup along the rows and columns might be the fuel mixture setting for a particular RPM and throttle position – I’ve made this map up at the moment, but you hopefully get the idea.

Example of how RPM and Throttle Position might map to Fuel Mixture

That type of map is a two dimensional  lookup. But this is where things get a little more complicated, MBE seems to call these 3D maps. And you can see why I guess. We can draw our rows and columns for the inputs and then draw the result as height above the floor of the map. Sort of like making a 3D map of a mountain.

There are other types of lookup, for instance 1D and 2D. I’m still not entirely sure what MBE mean by 1D maps… and how do they differ from the input data from the sensors but I’ll update this post when I figure that out.

A 2D mapping is a lookup table where there’s one input and one output, which might look a bit like this, you read the input along the bottom and then find the output by going up until you reach the line:

2D Map takes Input and looks-up the Ouput

Interpolation

Another critical part in how ECU’s use maps is called Interpolation. Interpolation is used to fill in “holes” in the map.

As we’ve seen, a lookup table allows you to take some inputs and “map” them to an output. And you can read across each axis of the lookup table to get to an output. But for some inputs there would be thousands of rows of data to lookup and the table would become huge. For instance, with RPM, the car can go from 0 to 7900 RPM, that would need 7900 rows on the lookup table. To cut down on the amount of memory needed to store these lookup tables in the ECU, the tables are what we call, sparsely populated, meaning that not all the rows or columns are laid out in the table.

Lets take a look at another example to explain…

Example of a sparsely populated lookup table

As you can see in the table above, we have inputs going down the left of 10 to 80 and inputs along the top also going from 10 to 80. It’s simple to work out the lookup for an input of 10 and 30, the answer is 300 (this is also a times-table).

But what if we have inputs of 25 and 55 for instance. We can’t just go to row 25 and column 55 because they’re not listed in our table.

We need to use interpolation.

There are a few types of interpolation we could use, but we’re going to stick to the simplest here – linear interpolation.

To do the mathematics behind the interpolation we need the 4 numbers around where we’re doing the lookup. The row we’re interested in is 25, so we need to take values around that, ie. row 20 and row 30. Similarly we’re looking for column 55 so we’ll take value from column 50 and 60.

So, we’re interested in the following data in green:

We’re interpolating 25,55 from rows 20 and 30 along with columns 50 and 60

In linear interpolation we assume the jumps between rows and columns is linear, i.e. the graph you draw between adjacent rows and columns is a straight line.

I’m not going to go into the whole maths behind how this would work, but for our example 25 is half way between 20 and 30, and 55 is half way between 50 and 60. That means we can take the four numbers in green and average them:

Output = (1000 + 1200 + 1500 + 1800) / 4 = 1375

We’ve now managed to get an answer from our lookup table for our question (what’s 25 x 55) without needing to have a table that’s 80 columns wide by 80 rows tall. That’s interpolation. And that’s how ECU’s store their maps in memory – they use sparsely populated maps and the micro-processor in the ECU does a lookup followed by the interpolation maths to come up with a mapped answer.

If you want the whole maths behind how that works then wikipedia has a write up on it.

Back to EC2 Files

The EC2 file that we’ll look at a bit more in another post has the following number of maps defined:

  • 1037 : 1D-maps
  • 137 : 2D-maps
  • 67 : 3D-maps

These map configurations in the Easimap EC2 files are just the configuration parameters of the maps, not the data inside the map, and I suspect that nowhere near those 1037+137+67 maps are used in our cars, but that needs more work to figure out how MBE construct the map definitions in the ec2 files.

At the moment there doesn’t seem to be any way of extracting a map from a locked 9A4 ECU. In theory we can build the map by filling in the data as we see the inputs and outputs of each map change. i.e. we can run the car and watch the throttle angle and engine speed then see how the ECU sets the mixture, as all those values change we can build the lookup table.

I’m hoping that I can get a Map file from someone with an unlocked ECU, then I can see if some of my theories about how the data is structure in a Map hold true.

Next we’ll get into the Easimap EC2 files a bit more and extract some data from them to automate the way we can access our ECUs.

ECU Diagnostics – part 6 : Reading Material

Another quick one, really quick this time…

Mark (CTRMint) had been doing his research and had told us about a book he was buying, and I just couldn’t resist…

The Car Hacker’s Handbook

The book isn’t the cheapest of purchases, at least not in the UK. But for me it was well worth it.

It took an evening to read the most interesting stuff, which was the first 2/3 of the book for me.

There was some great stuff in there and it really helped frame the issues we were facing. It introduced me to the idea that there might be other protocols on top of the CAN bus protocol, like ISO-15765 (ISOTP) and Unified Diagnostic Services (UDS). It also confirmed some things we’d come to think were probable, like the request/response procedure that devices seemed to be using.

It’s always difficult to think back about what value books like this were to us in our investigations, but I think it moved me along at least a couple of weeks of regular surfing in the space of an evening.

If you’re interested in picking apart a CAR diagnostics network then this book would be a great starter. If you’re an old hand at this sort of stuff then I suspect it would be too simplistic for you, but it was a great primer for me.

ECU Diagnostics – part 5 : The Correlator Dead-end

Hopefully this is a quick post…

The story so far is that:

  1. We’ve got a test bench made of a Raspberry Pi, CAN bus decoder board and an OBD Y-cable
  2. The Raspberry Pi can “sniff” all the packets sent between the car’s ECU and something that can talk to the car, like Easimap or an OBD scanner
  3. We can decode the CAN bus packets captured in the last bullet point and view them using software such as Wireshark
  4. We knew that the MBE ECU sends continuous data on the CAN bus (I call the protocol “MBE-Broadcast”) but there’s not much real data in those packets
  5. We know that the MBE ECU can talk the standards based protocol of OBD-II, but again there’s not much more than simple data there either
  6. SBD provides the Easimap software and USB to CAN bus adapter (MBE 985) that can get all the data from the ECU, but we don’t know how it does that

Easimap EC2

The Easimap Windows software app can get all the data it needs from the ECU. The app is designed to work with all of MBE’s ECU’s… not just the Caterham 9A4 but all the other ECU’s they make and covering all (probably) of their past ECUs.

The flexibility that these diverse ECU requirement brings, is achieved by having configuration files that Easimap loads once it’s decided which ECU it is talking to.

And those configuration files are loaded from a standard location after the software is installed and have an .ec2 extension. You can find the ec2 files in the Projects directory that Easimap installs.

When Easimap sees the Caterham 9A4 ECU it loads an EC2 file called:

9A4be52a.ec2

This file is not for the faint hearted. It’s a text file, but over 25,000 lines long. There are multiple sections to the file and it all cross references itself. I’ll talk about EC2 files and how I pulled out what I needed in a future post.

But for the moment, and at this stage of the testing, I was interested in the way the basic data definitions worked in the ec2 files.

A Simple EC2 Data Variable Example

For instance the Engine RPM data variable (RT_ENGINESPEED) is defined in the ec2 file like this:

[RT_ENGINESPEED]
Number of Dimensions = 0
Page = F8
Address = 237C
Bytes per Cell = 2
0 = SCALE_ENGINESPEED

The files are formatted like the Windows config file format, but it’s multi-layered and causes standard parsers to barf, but we can see basically what’s going on.

I think the RT in RT_ENGINESPEED means “real time”. But that’s just a guess.

Here’s what I think the rest means:

  • Number of Dimensions: This defines how many axies the variable has. Zero dimensions means its a single data item variable, containing a single data item. If it was a 1 dimension then it would be a simple one-dimensional array – containing a list of numbers. 2 dimensions is a 2D array with two axies. At least that’s my thinking at the moment.
  • Page: In computer speak we often talk about pages of memory. This “Page = F8” refers to a “page number” in hexadecimal of 0xf8.  In the old days CPU’s couldn’t deal with the large amounts of memory that were required of them and they had to load pages of memory at at time into the main processor memory (cache). These days that concept is not quite as prevalent, even though the MMU (Memory Management Units) of modern CPU’s still have throw backs to this system. Anyway, a page is a block of memory and each “page number” refers to that block. 
  • Address: This is also a reference to a computer memory location. It is the “index” into the page in hexadecimal – in this case 0x237c. We call it an index because it indexes the memory location in the page, like the index of a book it points to a memory location in the block we accessed with the page number.
  • Bytes per cell: Is telling us the data variable can be a maximum of 2 bytes, which is 16 bits and can take a value from 0 to 65,535. The ec2 file also defines 1, 3 and 4 byte variables.
  • 0 = SCALE_ENGINESPEED: This line is a bit more complex and I’ll leave its discussion to another post.

Right… so the reason I’m getting into this here is that the page and address parameters in this variable definition are what intrigued me. If Easimap was using the ec2 files as a definition of how it was going to talk to the car then the page and address parameters seemed to be very important.

Now back to the packets on the CAN bus.

Some CAN bus Easimap data

So, we knew nothing more about how Easimap was talking to the ECU on the CAN bus. The first packet captures we took from the car just showed us data like the following:

#  Time        Protocol    ID         Data
1 0.000000000 CAN 32 XTD: 0x0cbe1101 03 04 00 0d ef fb ff f7
2 0.000990262 CAN 32 XTD: 0x0cbe0111 10 0d e4 00 0d 23 39 41
3 0.001320498 CAN 32 XTD: 0x0cbe0111 21 34 62 65 35 33 30 00
4 0.058836571 CAN 32 XTD: 0x0cbe1101 03 04 00 5e ef fb ff f7
5 0.059825611 CAN 32 XTD: 0x0cbe0111 07 e4 00 5e 2b 07 01 00
6 0.506864643 CAN 32 XTD: 0x0cbe1101 10 0a 01 00 00 00 00 12
7 0.507713945 CAN 32 XTD: 0x0cbe1101 21 66 67 a8 a9 00 00 12
8 0.508917926 CAN 32 XTD: 0x0cbe0111 05 81 aa aa 16 00 12 66
9 0.517519386 CAN 32 XTD: 0x0cbe1101 10 09 01 00 00 00 00 1a
10 0.518228338 CAN 32 XTD: 0x0cbe1101 21 52 5c 5d 00 00 00 1a
11 0.519227915 CAN 32 XTD: 0x0cbe0111 04 81 84 00 00 00 1a 52
12 0.530752292 CAN 32 XTD: 0x0cbe1101 10 0a 01 00 00 00 00 e2
13 0.531488040 CAN 32 XTD: 0x0cbe1101 21 cc cd ce cf 00 00 e2
14 0.532530228 CAN 32 XTD: 0x0cbe0111 05 81 ff ff ff 07 e2 cc
15 0.545266012 CAN 32 XTD: 0x0cbe1101 10 23 01 00 00 00 00 f8
16 0.546032834 CAN 32 XTD: 0x0cbe1101 21 30 31 36 37 44 45 4c
17 0.546786859 CAN 32 XTD: 0x0cbe1101 22 4d 4e 4f 50 51 5a 5b
18 0.547536866 CAN 32 XTD: 0x0cbe1101 23 5c 5d 64 6a 6b 7c 7d
19 0.548296910 CAN 32 XTD: 0x0cbe1101 24 9e 9f a0 a1 d8 d9 da
20 0.549005102 CAN 32 XTD: 0x0cbe1101 25 db 9f a0 a1 d8 d9 da

It was at this point that we figured out that Easimap was using a request/response protocol. It was sending data to the ECU with an ID of 0x0cbe1101 and was getting data back from the ECU with an ID of 0x0cbe0111. You can see the changing ID’s in the packets above, if the line has an 0x0cbe1101 in it then it means Easimap is sending that data to the ECU and vice versa for 0x0cbe0111.

Now back to the pages and the address…

It seemed to me, that if Easimap was so configurable and the ec2 files defined the configuration, then the page and addresses were probably being sent to the ECU in some form. And that meant that they would probably show up in the “data” we were seeing on the CAN bus.

In the packets just above, I knew that Easimap was asking the car for Engine RPM (RT_ENGINESPEED) and that had a page of 0xf8 and an address of 0x237c from the ec2 file.

But I couldn’t see the two bytes of 0x237c in any of the data to or from the car.

Bit Shifting

It then seemed that if we couldn’t see the 0x237c in the data stream, and we were sure that Easimap must be sending it somehow to the car, then perhaps the address was being sent with a “bit shift”.

The CAN bus protocol itself is a very non-byte aligned protocol. What I mean by that is that the CAN bus protocol uses transmissions of things like the ID, CRC, start and stop bits that do not align to byte boundaries – they don’t align to the zero’th bit of 8 bit (byte) chunks.

So, perhaps the data being sent to and from the ECU was not “byte aligned” either.

I needed a correlator.

Correlator

If the address 0x237c was being sent to the ECU non-byte-aligned then I needed to look for its “bit pattern” in the data at non-byte-aligned positions.

I was therefore going to have to look for 0x237c in binary, which is

0010001101111100

And I was going to have to look for this bit pattern in the data. For example the first line of data in the capture above is:

03 04 00 0d ef fb ff f7

Which is this in binary:

0000 0011 0000 0100 0000 0000 0000 1101 1110 1111

 And so I would have to look for the first binary pattern (0010001101111100) within the second pattern, and then do that through the whole of the packet capture (which was about 3000 packets in total).

I needed a program… and in digital signal processing (DSP) terms this program had to correlate the address with the data.

So, I wrote this program here. And here’s some example output from it…

Bitwise search for: 0x237c
Row 1, Data:0x(ff95000000000000), 0b(1111111110010101000000000000000000000000000000000000000000000000)
Row 2, Data:0x(0495ff2800000000), 0b(0000010010010101111111110010100000000000000000000000000000000000)
Row 3, Data:0x(0327001d001d0000), 0b(0000001100100111000000000001110100000000000111010000000000000000)
Row 4, Data:0x(02aaaa008fd60000), 0b(0000001010101010101010100000000010001111110101100000000000000000)
Row 5, Data:0x(0152000039009549), 0b(0000000101010010000000000000000000111001000000001001010101001001)
Row 6, Data:0x(ff00000000000000), 0b(1111111100000000000000000000000000000000000000000000000000000000)
Row 7, Data:0x(ff00000000000000), 0b(1111111100000000000000000000000000000000000000000000000000000000)
Row 8, Data:0x(ff49000000000000), 0b(1111111101001001000000000000000000000000000000000000000000000000)
Row 9, Data:0x(ff95000000000000), 0b(1111111110010101000000000000000000000000000000000000000000000000)

Unfortunately I didn’t find anything – a dead-end. I couldn’t find any of the addresses from the ec2 file within the data I was capturing from Easimap talking to the car.

If my program had found something then it would have printed something out like this:

Bitwise search for: 0xff28
Row 1, Data:0x(ff95000000000000), 0b(1111111110010101000000000000000000000000000000000000000000000000)
Row 2, Data:0x(0495ff2800000000), 0b(0000010010010101111111110010100000000000000000000000000000000000)
Found something:
Row/Bit: 2/32, DW: 0x(ff28), DW: 0b(1111111100101000), Data: 0x(0495ff2800000000), Data: 0b(0000010010010101111111110010100000000000000000000000000000000000), SHIFTED: ff28

But it didn’t find anything other than the test data I had plugging into it, like that 0xff28 test above.

If I write the output of that test out a bit differently, you’ll see how the bits line up, we were looking for the first line, in the second line and we found it at bit position 16…

Found:
................1111111100101000................................
In:
0000010010010101111111110010100000000000000000000000000000000000

I’ve been involved in a number of attempts over the years to decode different communications protocols and these dead-ends happen a lot.

Ok, so we need to think about another way of decoding the protocol. 

ECU Diagnostics – part 4 : Wireshark Patching and OBD-II Results

When Wireshark first appeared, it was called Ethereal… and its motto was:

“Sniffing the glue that holds the internet together”

Wireshark sniffs the packet-glue that holds the internet together and is one of the most influential pieces of open source software there is. 

That’s a pretty bold statement and I even count projects like Linux itself in that comparison. It’s certainly up there.

Wireshark takes packets of data sent on a Network and converts the binary transmission into something us humans can understand.

It can take something like this data on a CAN bus:

3031363744454c4d4e4f50515a5b5c5d646a6b7c7d9e9fa0a1d8d9dadb

… and turn it into something like this that we can understand:

RT_COOLANTTEMP1(LIM)  ( Coolant Temp ) = 23.048 C

Wireshark, and its sister tools, have facilitated the democratisation of internet development. Without it, and in the early days of the internet, people would have to write their own test and development software or pay large sums of money for specialist software. Wireshark has allowed anybody, whether privately or commercially, to peek, ponder and pontificate about how all sorts of devices and software communicate with each other.

These days it is also relatively simple to add new functionality to Wireshark and allow it to decode and dissect any new protocol that may care to dream up. 

The only caveat is that the communication protocol in question needs to be packet based. i.e. it needs to happen in chunks of data. But, as it happens, that’s ok… Even the streams of data that we, as humans, perceive as being continuous are transmitted in chunks (packets) over the internet. Speech, Music, Video, Financial Transactions and plain old web-browsing are all transmitted over the internet in packets. And until recently they couldn’t hide from Wireshark.

However, more recently, much of the internet has moved to using encrypted communications. This is a lot more difficult for Wireshark to peer into, but for software developers (and for that matter hackers) that have the appropriate encryption keys, Wireshark is still an invaluable tool to write, debug and help deliver their software.

Dissectors

Wireshark is essentially a “frame-work”. By which we mean it provides a skeleton that support the real work of the software. Just like a human skeleton (frame work) provides support for the rest our body’s organs. The frame-work provides all the housekeeping functions,  like accessing the network, opening and closing files and displaying packets to a user.

That leaves the bulk of the “real work” inside Wireshark to software modules called “packet dissectors”. There is one or more packet dissector modules (a chunk of software) for each of the communications protocols that Wireshark can decode.

Remember that internet communications protocols are like different spoken languages. And each Wireshark dissector is like a translator for each of those different languages – turning binary data into human readable text.

 I just checked the Wireshark source code , and as I write this post in September of 2019, Wireshark has 1450 dissectors that ship with the software download. Each of those dissectors can turn a “machine readable” communications protocol into something that is “human readable” (see below for an example of that). And, of course, that doesn’t include the thousands of dissectors that people have written for themselves or their companies and that don’t get shipped with Wireshark as standard. I think that’s amazing.

So how does an internet packet sniffing application like Wireshark help with the poking around in the world of automotive communications?

SocketCAN

Each CAN bus “frame” of data (see ECU Diagnostics – part 2 : ECU, OBD and CAN for an explanation of CAN bus) can also be thought of as a “packet” of data. And it looks very much like any other internet packet based protocol. It has a start to the packet, a data section, flags (information) that tell us about the packet and error checking – all features of internet packets. The only real difference between a CAN bus frame/packet and internet packets is that it has an ID (identifier) that looks sort-of… but-not-quite like the source and destination addresses of an internet packet. But its a reasonable analogy and allows Wireshark to be able to work with CAN bus packets.

Enter SocketCAN. 

Because CAN bus frames are so much like an internet packet, we can convert them into something that Wireshark thinks is an internet packet and then let all the tools and software that are able work with internet packets do the hard work of dissecting CAN bus packets and displaying them for us.

SocketCAN takes a stream of CANbus frames and puts them into an internet packet format and then makes that packet stream available through a “socket”. It should perhaps be called Can-to-Socket instead of SocketCAN, but that’s quite a common trope for the internet… just to confuse everyone who are not in the know!

The internet runs on sockets… perhaps just like the power grids of the world. When a software application wants to access the internet, it plugs into a socket, just like you would if you want power for your electrical devices.. These are of course software sockets but the analogy is quite similar. In this example, the CAN bus frames are like the electricity that is fed to you through a power socket.

Internet sockets connect software running on a computer, or server, to another computer’s software that also use internet sockets to receive the data.

Now we have a way of taking the 1’s and 0’s on the CAN bus and turning them into a stream of packets that our computers can access through a socket interface!

SocketCAN is a Linux thing. It is actually a “kernel module”, which means it can run fast and with extra privileges that regular software doesn’t have. The real world upshot of this is that SocketCAN creates another network interface on the computer, just like a Wifi or Ethernet network connection… and means that CANbus frames drop into the computer just like they are coming from a Wifi of Ethernet connection.

Phew! We’re finally going to talk about looking at the data…

Capture Files

Wireshark showing CAN packets

Wireshark works with “captures”… it captures packets from a network and then displays them like in the picture above of a CAN bus capture. And just like Wireshark can capture packets from a Wifi or Ethernet connection to the internet, it can also now capture packets from our CAN bus, because we have SocketCAN turning the CAN bus frames into an internet packet for us.

An important point for us here is that Wireshark accesses these packets using an internet socket… just the sort of socket that SocetCAN will take CAN bus frames and convert them into internet sockets.

It can either view the packets “live”, meaning they appear on the Wireshark display as they are transmitted on the CAN bus, or it can show packets that have been “captured” to a file and then display them later by loading the file into Wireshark again. The computer file that this creates is called a “capture file” and it contains the digital frames taken from the CAN bus that SocketCAN turned into packets for us.

So what do we do with all of this?

First Captures of What our Cars are Talking About

So, after connecting our test rig up as in the previous post, we needed to see what the car was talking about on it’s CAN bus. We knew very little about how the car could communicate with us at this stage, so I broke out an OBD Scanner…

AutoLink OBD Scanner

At this point in the proceedings I remembered that in my box of tricks I had bought an OBD scanner. These scanners plug into the OBD port under the car’s dash and show whatever standard data requests the car can support.

I have the Autolink AL319 that I’d used when I built the car to see that everything looked ok with the engine management side of things. The AL 319 can decode a number of different, common, protocols. It turned out that all it could talk to the Caterham with was the standard-fast OBD-II protocol.

So I hooked up the Scanner to the car where the Easimap software had been and set it to check on the data from the car while I captured the packets on the Raspberry Pi. 

The main reason behind using the scanner was to see if it could probe the car in any ways similar to Easimap. Perhaps it would give us a clue about what other protocols the car could support.

The scanner was able to extract something like 15 data points (see the end of this post for a full list) and looked like this:

AutoLink AL319 OBD Scanner

To get the data captures you’ll see below I followed the test setup I had in the previous post… namely… Raspberry Pi was connected in parallel with the scanner and car and was running a tshark capture to a file.

Once I had powered on the car, the scanner comes to life (powered by the OBD port of the car) and starts to talk to the car. After it had “discovered” the car talking fast, extended, OBD-II then I was able to scroll through the various data screens while the Raspberry Pi captured the conversation to a packet capture file. 

So I opened the packet capture up in Wireshark expecting to see lots of data, but nothing really appeared.

It’s not particularly unusual for Wireshark to have problems dissecting packets. It can be a bit buggy sometimes. But I had hoped it would do better. In theory Wireshark has an OBD-II dissector. But all I could see was the “raw” CAN packets, as in shown in the picture below.

Something’s Not Working with the OBD-II Dissector

Wireshark trying to decode OBD-II protocol from scanner

If you look at the screen capture of Wireshark above, you’ll see that the first line (packet 142) shows the Protocol as OBD-II. But the remaining packets are not decoded correctly. From packets 200 to 219 we can see the Scanner trying different protocols on the car to see if it will respond… it finally does in packet 269, but Wireshark should be doing a better job of decoding from 269 onwards and displaying the OBD-II data, not just the hexadecimal format you can see above.

On further investigation, over a few hours and after eventually getting into the Wireshark source code… the problem was that the Wireshark OBD-II dissector was only able to decode packets with 11bit CAN ID’s (which is why it can decode packet 142 – it’s an 11bit ID the scanner first sends to the car).. And our cars are using 29bit CAN ID’s, which it turns out Wireshark couldn’t decode.

Bummer!

Fixing the Dissector

So, I rolled up my sleeves and set about fixing Wireshark’s OBD-II dissector.

That’s the beauty of open-source software. You download the code, fix what needs to be fixed… and if you’re lucky you can submit your fix back to the project and everyone else gets to benefit from your fix.

In the end I changed, or added, 95 lines of code to packet-obd-ii.c file in the Wireshark source code. Over the course of a few days the patch was reviewed, tested and eventually accepted by the project co-ordinators. So, hopefully as of the next public release (version 3.0.4) of Wireshark it should be able to dissect OBD-II communications to and from a MBE 9A4 ECU.

Woot woot!

The patch and all its gory details can be found here:

https://code.wireshark.org/review/#/c/34297/

With the patch installed the same capture file shown below…

CAN bus capture with OBD-II dissector not working

… now looks like this one below…

CAN bus capture with the OBD-II dissector now working

All the CAN bus byte data has been turned into something we can read more easily. You can see commands are sent from the scanner to the car and a response comes back. So, for instance in packet number 1948 the scanner sends the “calculated engine load” command and the car responds in packet 1949 with 0.00% (i.e. the car isn’t running).

But not only that.. because the Wireshark application and the Tshark command line tool both share the same code base, it means I can run a command like this..

tshark -i can0 -Y "(can.id != 0x0cbb0001)" -d can.subdissector=obd-ii

and get a simple text output of what the Scanner was doing:

#    Time         ID         Info
1891 18.384234117 [18db33f1] Freeze DTC
1892 18.386480284 [18daf101] Freeze DTC: < 22 26 >
1948 18.932752338 [18db33f1] Calculated engine load
1949 18.933708266 [18daf101] Calculated engine load: 0.00 %
2004 19.480255834 [18db33f1] Engine coolant temperature
2006 19.482689349 [18daf101] Engine coolant temperature: 21 °C
2061 20.029324990 [18db33f1] Intake manifold absolute pressure
2062 20.031880206 [18daf101] Intake manifold absolute pressure: 200 kPa
2118 20.578419385 [18db33f1] Engine RPM
2119 20.581109100 [18daf101] Engine RPM: 0.00 rpm
2175 21.127303969 [18db33f1] Timing advance
2176 21.128371192 [18daf101] Timing advance: 0.00 °BTDC
2231 21.674841186 [18db33f1] Intake air temperature
2233 21.676107628 [18daf101] Intake air temperature: 16 °C
2288 22.222260164 [18db33f1] Throttle position
2289 22.224313834 [18daf101] Throttle position: 22.35 %
2345 22.771129266 [18db33f1] OBD standards
2346 22.773691168 [18daf101] OBD standards: EOBD
2402 23.320629505 [18db33f1] Oxygen sensors present (4 banks)
2403 23.323019928 [18daf101] Oxygen sensors present (4 banks): Bank1 sensors: 1 , Bank2 sensors: None, Bank3 sensors: None, Bank4 sensors: None
2459 23.869724288 [18db33f1] Fuel Rail Gauge Pressure
2460 23.872168914 [18daf101] Fuel Rail Gauge Pressure: 2400 kPa
2515 24.419230119 [18db33f1] Absolute Barometric Pressure
2517 24.421439008 [18daf101] Absolute Barometric Pressure: 0 kPa
2572 24.968236310 [18db33f1] Control module voltage
2573 24.969567270 [18daf101] Control module voltage: 11.693 V
2629 25.516453162 [18db33f1] Commanded throttle actuator
2630 25.518540406 [18daf101] Commanded throttle actuator: 00

If you want to see how I came up with that tshark command line options then you can find more in the tshark man pages here. But simply put the options translate as:

  • -i tells tshark to use the can0, SocketCAN network interface
  • -Y option restricts the displayed packets so we don’t see the MBE-Broadcast frames that use the CAN bus ID of 0x0cbb0001
  • -d is the important one that tells tshark to take any CAN packets (using the can.subdissector) and decodes them as OBD-II

For those of you that like your graphical user interfaces, that last dump of text may seem like a step backwards. But the advantage of getting a text output from a packet capture is that you can search and analyse the capture more easily. Wireshark allows you to do some simple captures and filters, but once I have a textual version of the capture I can more easily just look for things like engine RPM, or coolant temperature. It would be nice if Wireshark could do more of that, but that’s not something I want to fix at the moment.

And here’s the full list of what data an MBE 9A4 Caterham stock ECU can report via the OBD-II protocol:

  • Monitor status
  • Freeze DTC
  • Calculated engine load
  • Engine coolant temperature
  • Intake manifold absolute pressure
  • Engine RPM
  • Timing advance
  • Intake air temperature
  • Throttle position
  • OBD standards
  • Oxygen sensors present (4 banks)
  • Fuel Rail Gauge Pressure
  • Absolute Barometric Pressure
  • Control module voltage
  • Commanded throttle actuator

So there we are. I started off thinking that pulling the scanner out of my toy box would be a 5 minute investigation. A week later I had the analysis I wanted but I also got to make my first patch of Wireshark and, after using the software for 20 years, I’d got a better understanding how it really works.

Now we’ve seen what the car can do when it talks OBD-II, we can get back to understanding the Easimap protocol some more.

ECU Diagnostics – part 3 : Test Setup

Ok, so we have an OBD port/connector on our cars that connects directly to the ECU. How are we going to figure out how to talk the right communications protocols to the car to get at its internal data?

We need a test rig to do some experiments with.

If you're looking for a post about "How to..." set all this up then that will be coming at the end of this series of posts in a "How to set up a Raspberry Pi to talk to a Caterham" post

After a bit of research over the years I’d already purchased a few bits and pieces to connect to a CAN bus. There are many ways of doing this but I settled on the following approach from my box of bits and pieces.

Here’s a schematic view of the setup…

Caterham ECU Diagnostic Test Setup

And then this is what it looked like in real life, Easimap is running in the top left corner on the screen of the MacBook. The Raspberry Pi and PiCAN are propped against the front left of the laptop.

OBD Test Setup. Raspberry Pi with PiCAN2, MacBook running Windows and Easimap+MBE985, connected to car through OBD Y-Cable

So what is all that doing? Firsly, lets give a quick run-down on all the bits and pieces:

Raspberry Pi

Raspberry Pi 4

Raspberry Pi: This is a Raspberry Pi 4, but anything from a Raspberry Pi 2 onwards is probably ok. Certainly a 3 or 4 are great. The Raspberry Pi is running Linux and can interface to the PiCAN to extract CAN bus frames.

PiCAN

PiCAN3 sitting on top of a Raspberry Pi 4

PiCAN: Is an interface card that plugs on top of the Raspberry Pi. The PiCAN2 SMPS (switched mode power supply) can also provide power from the CANbus to the PI. If you have a Pi4 and want it to be powered from the CANbus then you’ll need the PiCAN3.

Wireshark

Wireshark showing CAN packets

Wireshark/Tshark: Wireshark is a Network Protocol Analyzer or packet sniffer application that can intercept packets on a network and decode them into a human readable form. It runs on Windows, MacOS and Linux and can decode, or dissect as its known, hundreds of different network protocols and is immensely useful in today’s world of computing. Tshark is the command line version of Wireshark (built from the same source code) and is what I used in this test setup. I used tshark because I was running my Raspberry Pi with no graphical user interface, just remote command line access using ssh from my MacBook. Tshark allows you to capture and view packets just like Wireshark but without the graphical interface.

We’ll be talking about Wireshark in a future post as it soon became apparent that Wireshark wasn’t properly able to decode the CAN bus communications from my car, but more on that in the next post!

Easimap

Easimap – MBE ECU Windows Diagnostic Application

Easimap: A windows application from SBD that can interface with a 9A4 ECU and extract data. For unlocked ECU’s (not the standard Caterham one) it can also extract/load Maps, and download/upload ECU images (the whole software and data setup of the ECU). Easimap can graphically show ECU data in real-time. Easimap runs on Windows but in this test I was running Windows in a Virtual Machine on a MacBook Pro.

MBE985

MBE 985 CANbus to USB Interface

MBE985: This is an interface device that converts CANbus data into a USB format that a Windows PC can access using Easimap. Easimap sends commands to the car and receives data back by communicating through the MBE985 over USB which then sends the communications over the CAN bus.

For this of a curious nature, I pulled my MBE 985 apart to see what made it tick. We were wondering if there was much translation going on between what we had seen on the USB interface and what was being sent to the CAN bus. As you can see from the picture below, the MBE 985 is a reasonably complicated interface – more than you might expect in a CAN bus to USB interface.

MBE 985 internals

The interface is using a 25Mhz Infineon processor that has a CAN bus interface built in. It also has a 4Mb Flash chip on the board. So this interface is doing some reasonably serious processing!

OBD Y-Cable

OBD Y-Cable

OBD Y-Cable: In my setup I used this Y-cable to connect everything together. It allows the Easimap/MBE985 combo to connect to the ECU but also allows the Raspberry Pi/ PiCAN to “see” the communication between Easimap and the ECU – it allows the Raspberry Pi to “sniff” the CANbus traffic. Another way of putting it is to say that the Easimap and Raspberry Pi are “in parallel” with each other and both able to see the communications to the car at the same time. The MBE 985, PiCAN and ECU are now three devices on the CAN bus.

MacBook Pro

MacBook Pro Laptop: I used a MacBook as the host to run the Easimap Software but which actually only runs on Windows. And in order to get around that small niggle,  I had to use Parallels Virtual Machine software to run Windows on the Mac as a “Virtual Machine Host” and then to run Easimap inside that Windows virtual machine. If MS Windows it your thing then you could replace this Mac/Parallels/Windows/Easimap configuration with a PC/Windows/Easimap setup.

Parallels

Parallels Virtual Machine Software for Mac

Parallels:  This is a Virtual Machine application that allows a Mac to run other operating systems “on top” of MacOS. It can run Windows, Linux and even another copy of MacOS. You don’t need this if you’re running Windows on the bare metal (i.e. Windows directly on the computer).

With all that connected up we can start to investigate how Easimap talks to the car’s ECU.

Capturing Packets

I guess there are a number of ways of figuring out what Easimap was doing. For instance, we could have had  go at disassembling the Easimap software… But that would have been a massive task and very unlikely to yield any results in a non-geological time frame. I’ve disassembled other people’s code in the past… and it took months.

The best approach was probably going to be a technique called “packet sniffing”. And we had two points of entry there too to think about.

One approach that was tried was to “sniff” the packets being sent by Easimap to the MBE985. That would mean sniffing the USB connection between Easimap and the MBE985 and Wireshark is a great way of doing that. James (Aerobod) on Blatchat had been doing this and got some good background info…

But that wasn’t going to be my approach…

I wanted to get direct access to the CANbus. I wanted to see what I would have to signal on the CAN bus itself if I was going to replace Easimap. That meant sniffing the CAN bus frames and pulling apart the communications at that level.

And now to some Packet Sniffing

Ok, so now we have our hardware and software all set up to capture some of the communication between Easimap and the car.

The process will be to:

  • Connect all the hardware and turn it on
  • Run the tshark packet capture on the Raspberry Pi
  • Start up Easimap on the Windows Virtual Machine
  • Now turn on the car (my car has a battery isolator) and then put the car into ignition switch position 2 – that seems to keep the ECU talking
  • Stop the capture and transfer the capture file (.pcapng) over to my Mac for review

I spent about 10 minutes one evening taking a few initial captures, and those files kept me going for over a week. That’s often the way with these things. There’s so much to learn in the initial investigation, that you spend ages looking at just a few seconds worth of data.

Once I had the captures on my laptop I could review and anaylize them with the car and the Raspberry Pi off.

Initial Results – MBE-Broadcast

The CANbus frames I’m going to show now just contain the ID and data parts. The captures also contain the error, extension, etc flags that we talked about in a previous post… but there’s nothing interesting in those bits of the frames now we know we have some good communication between PiCAN2 and the car, so I won’t show them here.

To get a capture on the raspberry Pi we just run the following command. There’s a whole setup process for the Pi that I’ll talk about in its own post and put a link [here] when it’s done. But once that’s done we just run this command.

-> sudo tshark -i can0

And if we want to capture the packets to a file then we could supply a file like this:

-> sudo tshark -i can0 -w /tmp/mycapture.pcapng

When the car is first turned on it starts churning out the following packets:

# Time        ID          Data
8 0.069921880 0x0cbb0001 04 95 ff 28 00 00 00 00
9 0.079854830 0x0cbb0001 03 27 00 1e 00 1e 00 00
10 0.089837445 0x0cbb0001 02 aa aa 00 af d6 00 00
11 0.100195073 0x0cbb0001 01 50 00 00 39 00 95 48
12 0.110361352 0x0cbb0001 ff 00 00 00 00 00 00 00
13 0.119903104 0x0cbb0001 ff 00 00 00 00 00 00 00
14 0.129903793 0x0cbb0001 ff 48 00 00 00 00 00 00
15 0.139997221 0x0cbb0001 ff 95 00 00 00 00 00 00

Those 8 packets get repeated continuously… non-stop… without the Easimap software running. 

A quick description of what we’re seeing here, the columns of data represent:

  • # is packet number
  • Time is the number of seconds since the tshark capture started
  • ID is the CAN bus ID (arbitration field / address)
  • Data is the 8 bytes of data for each frame (in hexadecimal).

Not a lot of that made sense. And to honest we haven’t really dug into this data stream. My car’s engine wasn’t running here, so there’s probably even less of interest in those packets above. It seems to be something proprietary to MBE though and I suspect has some useful information in the stream but because the data is so limited we didn’t delve into it. I’ve called this data stream “MBE Simple” (because it comes from the MBE 9A4 ECU) and is characterised by the CANbus ID of 0x0cbb0001 and doesn’t have the same depth of info as the next protocol we’ll look at.

At first we thought these messages were all we were going to receive. There was talk on the forums that you needed to send an “unlock” frame to the car and then it would add more data to this Simple data stream. That turned out to be a red herring but initially we thought that the unlock code was in some way opening a CANbus bridge and we’d see more data once it was opened… that isn’t the case as we’ll see later.

We’ll come back to this protocol in a full post on the MBE Simple protocol later on.

Next Level of Results – MBE ISOTP

So, we’re not so interested in the MBE Simple protocol. Before we had Easimap running we did spend a couple of days trying to figure out if it was useful. But as soon as we got sniffing Easimap communications we knew we wanted to move on.

I’ve called this next protocol “MBE ISOTP” and we’ll have to wait for a future post on exactly why I’ve called it that, but suffice to say, this is the protocol that Easimap uses to talk to the car.

As soon as Easimap starts up, it begins to talk to the car… and we get this initial communication that doesn’t repeat (at least not for a long period after the car starts up):

#  Time        ID         Data
28 0.265548270 0x0cbe1101 03 04 00 0d 26 40 42 43
29 0.266543253 0x0cbe0111 10 0d e4 00 0d 23 39 41
30 0.266872637 0x0cbe0111 21 34 62 65 35 33 30 00
37 0.326052544 0x0cbe1101 03 04 00 5e 26 40 42 43
38 0.327045620 0x0cbe0111 07 e4 00 5e 2b 07 01 00

Then we get this set of 31 packets that repeat continually:

#   Time        ID         Data
80 0.736160089 0x0cbe1101 10 0a 01 00 00 00 00 12
81 0.736958836 0x0cbe1101 21 66 67 a8 a9 00 00 12
82 0.738230722 0x0cbe0111 05 81 aa aa 16 00 12 66
84 0.749027454 0x0cbe1101 10 09 01 00 00 00 00 1a
85 0.749733313 0x0cbe1101 21 52 5c 5d 00 00 00 1a
86 0.750875498 0x0cbe0111 04 81 84 00 00 00 1a 52
89 0.767924313 0x0cbe1101 10 0a 01 00 00 00 00 e2
90 0.768777632 0x0cbe1101 21 cc cd ce cf 00 00 e2
91 0.769825875 0x0cbe0111 05 81 ff ff ff 07 e2 cc
93 0.780880102 0x0cbe1101 10 23 01 00 00 00 00 f8
95 0.781565869 0x0cbe1101 21 30 31 36 37 44 45 4c
96 0.782301116 0x0cbe1101 22 4d 4e 4f 50 51 5a 5b
97 0.783061863 0x0cbe1101 23 5c 5d 64 6a 6b 7c 7d
98 0.783818702 0x0cbe1101 24 9e 9f a0 a1 d8 d9 da
99 0.784526524 0x0cbe1101 25 db 9f a0 a1 d8 d9 da
100 0.785768596 0x0cbe0111 10 1e 81 5e 39 cc 48 dc
101 0.786095776 0x0cbe0111 21 50 9c 89 1e 85 5e 39
102 0.786437863 0x0cbe0111 22 a8 0d 66 a0 00 b0 4f
103 0.786777913 0x0cbe0111 23 00 00 56 95 80 09 00
104 0.787102630 0x0cbe0111 24 80 00 80 a1 d8 d9 da
106 0.798260523 0x0cbe1101 10 0a 01 00 00 00 00 f9
107 0.799080454 0x0cbe1101 21 ba bb bc bd 00 00 f9
108 0.800070197 0x0cbe0111 05 81 78 1e dc 1e f9 ba
110 0.810033090 0x0cbe1101 10 09 01 00 00 00 00 fa
111 0.810853632 0x0cbe1101 21 64 65 6c 00 00 00 fa
113 0.812042780 0x0cbe0111 04 81 8c 1e 00 00 fa 64
115 0.821498922 0x0cbe1101 10 0e 01 00 00 00 00 fd
116 0.822377593 0x0cbe1101 21 20 24 25 26 40 42 43
117 0.823099192 0x0cbe1101 22 4d 24 25 26 40 42 43
118 0.824343987 0x0cbe0111 10 09 81 40 00 00 01 80
119 0.824665352 0x0cbe0111 21 b0 40 40 26 40 42 43

Ok, so we got a good dump of data from the car, but what does it all mean? We were baffled. You can see the packet numbers here aren’t contiguous… in-between these MBE ISOTP packets are the MBE Simple packets – interspersed – I’ve removed them here so you can see the “raw” MBE ISOTP packets and what we were now trying to decipher.

Those 31 repeating packets were all being sent and received within a 100ms time frame. And they repeat almost instantly. That implies that Easimap is updating its graphical display 10 times a second. At least that sets us a reasonable rate at which we might be able to do the same. A rev-counter display or something similar will be more than responsive enough if we can refresh it at that rate.

Now… Before we can move onto decoding this some more, we need to take a bit of a diversion back into Wireshark, because things weren’t adding up there at the moment.