ECU Diagnostics – part 2 : ECU, OBD and CAN

ECU

Caterhams, like all modern cars, have an Engine Control Unit (ECU) – a black box full of electronics, controlled by a microprocessor that manages how the engine runs.  And because its has a microprocessor, that means it also runs some software to control everything. It also connects to a bunch of sensors, like temperature, pressure, engine speed, air-flow, lambda etc which it then uses to set things like the engine’s ignition advance/retard, air/fuel mixture and other stuff.

The ECU on a factory, or self-build car is manufactured by MBE Systems and seems to be supplied to Caterham through SBDMotorsport in the UK. The model used by Caterham is the 9A4 that runs Caterham specific software and mapping. It is also “locked” – meaning that there is no user accessible way of modifying the software or the map. We’ll talk about maps and software in a later post.

What these MBE ECU’s do allow though, is access to the internal sensor information used by the ECU to run the engine. There are many, many parameters used by the software to make an engine run properly and we hoped to be able to see all this information in “real time” – meaning we can see the data change in front of our eyes, hopefully being updated many times per second.

It’s also important to understand that there is both primary data (taken directly from physical sensors) and derived data (secondary, extrapolated or calculated data derived from the primary data). We’ll hopefully be able to see both.

The standard location for the ECU is underneath the battery in the engine bay:

MBE 9A4 ECU located under the battery and behind the grey plug connecting it to the wiring loom

OBD

In order for us to be able to interrogate the ECU and extract all this data, we need some way of connecting a computer into the ECU. Handily, legislation has been in place for a number of years that requires car manufacturers to fit a car with a diagnostic connector. This is known as the OBD port and looks like this:

OBD connector located above the ignition switch on the steering column

It’s through this connector that we’ll be getting access to the ECU and hopefully seeing what’s going on. The OBD port on a Caterham is located above the steering column underneath the dash. On my 2017 car it is loosely mounted, dangling on a short length of electrical cables that run to the connector. So you need to be careful when handling it so you don’t pull any of the cables out of the connector.

There are a bunch of standards that define the physical and electrical OBD connector used on cars. What data that’s sent through this connector (I’ll also call it a port) is mostly open to the car manufacturer as to what gets sent and what standard transmission protocols are supported. It’s also very common that the data sent on these ports is proprietary to the particular manufacturer, each one having their own protocols.

For those reading this that aren’t quite so into communications protocols – a protocol is like a language. Computers and tech stuff communicate between each other using protocols. These protocols define what gets sent between two computers and how to interpret it. For instance, there are now over 8000 protocols that define how devices communicate over the internet, and more are being added every day. Check out the IETF RFC pages to learn more about those protocols and standards.

To make things more complicated, manufacturers of different equipment may interpret the protocol definitions (known as “standards”) differently and so end up with dialects of these languages. So the protocols differ from one manufacturer to another just like different dialects of French or English.

Fortunately, there is one protocol that does seem to be reasonably well followed by most car manufacturers and that gives us something to start with. That protocol is often known as OBD-II (OBD 2). We’ll go into more detail about the protocols supported by the MBU 9A4 in a future post.

As we started this investigation, learning about a Caterham’s ECU diagnostic port, we knew only a few things:

  • SBD Motorsport supplies a piece of Windows software (Easimap) that can get all the available data points from the car
  • OBD-II Scanners can access some data from the car (though not as much as Easimap)
  • People have tried to access the diagnostic data on a Caterham before and not been able to find much

So, we started with a general assumption that because the Easimap software could interrogate the car for lots of data, then hopefully we could too. Though one thing on our minds as we started out on this journey, was whether the data exchange (protocol) used between the car and the Easimap software was encrypted – that would cause a whole further level of investigation and might lead to all this effort being fruitless.

Early on in our investigations we wondered if Caterham and MBE were doing something special on the OBD port that was proprietary. However, a quick check under the dash and we can see that Caterhams have only 4 wires connected to the OBD port. There is 0v, 12v, CAN-L and CAN-H. CAN-L and CAN-H make up a twisted pair, see CAN bus below for more on those signals. So, none of the reserved connections on the OBD port were being used for anything special.

OBD Connector. 2 x 0v (black), 12v (purple), CAN-L and CAN-H

CAN Bus

There’s another twist to the OBD port. It’s one thing to know what it physically looks like and how to connect to it, but its another to be able to understand the data that’s flowing through the connector.

The first layer of that puzzle is to know that two of the twelve electrical connections on an OBD port support a standard called CAN bus. CAN stands for Controller Area Network and is used in industrial and automotive systems. It’s called a network because you can have many CAN bus devices connected in a network. In computing terms a “bus” is something that transfers data from one device in the network to another, we say that the “bus is shared” between the devices. Modern cars can have more than 50 devices in their CAN bus network; controlling everything from the engine, gearbox, infotainment systems, door locks, windows, lights, brakes and everything in between. My 420R 2017 Caterham has only one CAN bus device, the ECU. It seems that other cars (such as Sigma engined cars also have the rev-counter connected to the CAN bus – but not on my Duratec car).

Most modern cars also have multiple CAN busses: private ones for the safety critical components and public ones connected to the OBD port and more readily accessible to people like us. These private and public CAN busses are often connected together via a CAN bus bridge. There’s no evidence so far that there is any more than one CAN bus on a Caterham.

CAN bus defines both an electrical connection (voltages, termination impedances etc) but also a transmission protocol… i.e. what sequences of voltage changes mean (how fast is it, is it digital or analogue, is it serial or parallel communication ???).

So the CAN bus standard defines all this with the following key points for Caterham tinkerers:

  • There are two electrical signals making up a single serial data channel. The two signals are differentially coded, meaning you take the two electrical cables as a pair and use the voltage difference between the two to give you a single signal. The two signals are called CAN-L and CAN-H (CAN Low and CAN High).
  • When there’s no data being transferred on the bus the two data lines, CAN-L and CAN-H sit at about 2.5V. It’s one of the main ways you can find a CAN bus on a car… looks for pairs of wires that have a voltage of about 2.5V on them.
  • The two CAN-L and CAN-H signals combine to create a single serial data stream – meaning digital bits are transferred one at a time and successively.
  • CANBus can run up to 1Mbps (mega-bits-per-second). On a Caterham the speed is set to 500,000 bps.
  • There are three main types of CAN bus signalling. Standard addressing (using 11 bit identifiers (IDs)), Extended addressing (using 29bit IDs) and CAN-FD (or Flexible Data – sending larger amount of data in one go). Caterham’s use Extended addressing and not FD.
  • A stream of bits being sent on the bus forms a frame.
  • Each frame contains information about:
    • when the frame starts
    • what type of frame it is (standard, extended, error, etc)
    • how much data is in the frame (up to 8 bytes)
    • error checking
    • acknowledgement bits and
    • end bits
  • Data on the CAN bus can be either:
    • Broadcast: a device on the bus continuously sends data at pre-determined intervals. The 9A4 ECU does broadcast some information on the bus (I’ve called this protocol “MBE-Broadcast”) and I’ll talk about this in a future post
    • Request/Response: This is how the OBD-II protocol works and how Easimap communicates with the car. A device that want’s data (Easimap on a computer or a sniffer/scanner) “requests” data from the ECU and the ECU “responds” with the data. We’ll talk about the “OBD-II” and “MBE-ISOTP” protocol in later post.

For those interested, the format of a CAN bus packet is shown in the diagram below. Later on we’ll see that all we’re really interested in is the ID, Extended ID and the Data Field. The rest of the protocol is either handled by the CAN bus chipset or by the low level software (in my case that’s the Linux SocketCAN kernel driver).

Extended 29bit CAN bus packet format. From the Microchip MPC2515 datasheet. 

If that diagram isn’t making sense then perhaps this paragraph is for you: Each of the oblong boxes running across the centre of the diagram represents a single bit transmitted on the CAN bus. Therefore, time runs from left to right, so the leftmost oblong (bit) on the diagram is sent on the CANbus first, followed by the second, etc etc until the last bit represented by the rightmost oblong on the diagram is sent last. All those bits together are a CANbus frame and can contain up to 8 bytes of data on the MBE 9A4 CAN bus.

If you want more info on what all the stuff means in the diagram then I’d recommend heading over to the CANbus wikipedia link, I can’t add much more to what they say there.

There’ll be a post later in this series about me connecting my Logic Analyzer to the CAN bus and hopefully the screenshots there will explain a little more about how data is sent on CAN bus.

Arbitration Fields, CAN bus IDs and Addresses

Having said that, I hope I can clear up one thing that caused me confusion at first. The diagram above and the wikipedia page shows a field called the “Arbitration Field”.

At the physical level (i.e. the 0’s and 1’s on the electrical wires) the arbitration field controls who gets access to the bus. It does this because if two devices start to transmit on the bus at the same time then the one with the lowest numeric field code “wins” and continues to transmit, while the second device with the higher arbitration code backs off and tries again later. However, at a higher level, such as SocketCAN (see below), these arbitration fields are also like addresses, in computer terms, and different devices will listen for different arbitration fields which are known as ID’s (identifiers).

And for even higher software levels such as ISOTP (see a future post) they are also known as transmit and receive addresses. 

That’s all a little confusing, but all you really need to know is that Arbitration Fields, CAN bus ID’s and the ISOTP Receive and Transmit Addresses are all the same thing.

CAN bus Software

We’ll be talking some more about the CAN bus software in a future post but for the moment it’s worth pointing out that there seem to be a few ways of accessing CAN bus data for writing your own code on a computer.

I’m mainly interested in using Python as a programming language to do my tinkering and in my opinion the only way to do that is with a Unix derived system. And that means either Linux (RedHat, Ubuntu and their derivatives) or MacOS (BSD based).

… and for those operating systems and for Python there are a couple of ways of poking at the CAN bus.

  1. SocketCAN: this is the way I went. For me it gives closer access to the bus and allows me to use packet capture software like Wireshark. The CAN bus drivers are included in modern kernels and there is good support for other kernel mode protocols like ISOTP that we’ll need later. The CAN bus kernel module creates an interface that makes the CAN bus look like a network socket and so turns CAN bus frames into packets like an IP packet. These packets are then available to software applications like Wireshark to they can be captured and viewed. Python has a good wrapper for SocketCAN and is simply called python-can.
  2. ELM327: Another way to get CAN bus frames into a computer is to use a ELM327, or similar device. They can connect to phones and computers using Bluetooth or Wifi and have python support through the python-OBD project. They’re good because they mean your computer can be remote from the car. But I’m not a big fan of Wireless for critical connections and I’m hoping that my future OBD dashboard will be critical. They also have to work in “user space” (i.e. not in the kernel) and so can suffer from Linux’s susceptibility to not being as “real time” as some other OSes.

Yet another route for accessing CAN bus is through even smaller computing devices. I’ve spent a lot of time playing with the Arduino and Adafruit Feather devices over the years and I’m thinking that at some point I might switch from the Linux environment to Arduino for this project. These Arduino derivatives have their own programming environment and software drivers for things like CAN bus and they are also more “real time” than OSes like Linux and especially Windows. There are CANbus “shields” for Arduino and even Feather CANbus boards for the Adafruit feather line. But for the moment, Linux is a great way to figure out what’s going on with the car and to get rapid development done.

Example CAN packets

We’ll get into discussing the protocols used on a CAN bus in much more detail in furture posts, but for the moment here’s what three CAN bus frames look like to and from the car. The first two packets are to the car from Easimap and the third is the car’s response back to Easimap. The frames below are how Wireshark (a packet sniffer) displays the frames once captured from the CAN bus, it takes the serial stream of bits and decodes them into what each bit means, displaying them in a more “human readable” form…

Frame 626:
Controller Area Network
...0 1100 1011 1110 0001 0001 0000 0001 = Identifier: 0x0cbe1101
1... .... .... .... .... .... .... .... = Extended Flag: True
.0.. .... .... .... .... .... .... .... = Rem. Tx Req. Flag: False
..0. .... .... .... .... .... .... .... = Error Flag: False
Data: 10 0a 01 00 00 00 00 12

Frame 627:
Controller Area Network
...0 1100 1011 1110 0001 0001 0000 0001 = Identifier: 0x0cbe1101
1... .... .... .... .... .... .... .... = Extended Flag: True
.0.. .... .... .... .... .... .... .... = Rem. Tx Req. Flag: False
..0. .... .... .... .... .... .... .... = Error Flag: False
Data: 21 66 67 a8 a9 00 00 12

Frame 628:
Controller Area Network
...0 1100 1011 1110 0000 0001 0001 0001 = Identifier: 0x0cbe0111
1... .... .... .... .... .... .... .... = Extended Flag: True
.0.. .... .... .... .... .... .... .... = Rem. Tx Req. Flag: False
..0. .... .... .... .... .... .... .... = Error Flag: False
Data: 05 81 aa aa 16 00 12 66

CANbus packets will be printed differently depending on which software you use to look at them. In this particular example from Wireshark:

  • 0x0cbe1101 and 0x0cbe0111 are CAN bus identifiers (ID’s) or Arbitration fields written in hexadecimal (the 0x means hexadecimal). From the discussion above about arbitration, the device sending the ID of 0x0cbe0111 would take priority over the device sending 0x0cbe1101 . The first two frames with an ID of 0x0cbe1101 are sent from Easimap and the last frame with an ID of 0x0cbe0111 is the response from the car’s ECU
  • There are various flags shown, extended flag (meaning 29bit IDs), Remote Transmission Request Flag and the error flag.
  • The hexadecimal numbers on the Data line are the data. 

Well that’s about it for the basics of ECU’s, OBD and CAN bus. We’ll build on all of this in future posts.

ECU Diagnostics – part 1 : Introduction

I’d been planning to have a look at the car’s On Board Diagnostic (OBD) port for over a year, but hadn’t really got into the project. I’d bought the required cables and some test equipment, but other stuff always seemed to be higher priority. That was all about to change with a post from Mark (CtrMint) on Lotus7.club‘s Blatchat.

The thinking behind my project was to create a “device” that could plug into the diagnostic port on my car and provide a readout of what the car was thinking… perhaps that might turn into a simple project like change-lights or a race-computer – but a bit more high-tech than a few LEDs on a black box.

So that was my motivation, a home brew race-computer come dashboard come change-lights.

But back to Blatchat… Mark had asked for help looking into ECU Diagnostics and had started a long thread on what he thought was going on (here’s the thread link for club members). Seeing as it was something on my list too I thought I’d pitch in. I hadn’t appreciated how much I’d get sucked into the project, but now, about a month later and many, many hours of work I think I know enough to provide a few posts on the subject.

By the way, I know I’ve plugged Mark’s website before but he does a great job of blogging his Caterham life.

In all though, I think this is going to be a bit more than a few posts… we’ve been busy. It’ll be more like 15+ posts I think.

There’s a lot to talk about and I want to break it down so others can dip into the subjects and not have to plough through one monstrously long post. Some readers will want to get into all the nitty gritty and others will just want to browse. I’ll try and summarise things at the end for people who just want the conclusion.

We’ll be getting into the details of the protocols being used by the ECU, how we decoded those protocols, what we learnt, what software we used, and what software we had to change and/or write, and what we’re hoping to achieve with what we’ve learnt.

My intention with these posts is not to repeat what you can find in other places on the internet about car diagnostics. This is meant to be about diagnostics on a Caterham and where appropriate to point you at other resources to find out more detail, or even the basics. Hopefully it will pull a lot of things together to show what we did, how we did it and what it means for people wanting to find out more about Caterham ECU diagnostics. I hope that makes sense.

I’ve written the posts to be read by someone who is technically minded (aren’t all Caterham owners? 🙂 ) but who may not be steeped in internet knowledge. Hopefully those of you that are internet nerds won’t mind the extra words and can also find what you need without despairing at my efforts to keep everyone up to speed.

One final comment: THESE POSTS AND ESPECIALLY THE SOFTWARE I’VE MADE AVAILABLE HAVE NO WARRANTY OF ANY KIND.

So here we go for a marathon series of posts. I’ll update this page with links to the completed posts as they come along…

  1. Introduction
  2. ECUs, OBD and CAN
  3. Test Setup
  4. Wireshark Patching and OBD-II Results
  5. The Correlator Dead-End
  6. Reading Material
  7. ECU Maps and Mapping
  8. Easimap uses ISTOP (sort of)
  9. The Easimap Protocol Theory
  10. Decoding EC2 Files
  11. Logic Analyzer on a CAN bus
  12. OSI 7 Layers for Caterham Diagnostics
  13. Three Diagnostic Protocols in the MBE 9A4 ECU
    1. Protocol 1 – MBE-Broadcast
    2. Protocol 2 – OBD-II
    3. Protocol 3 – MBE-ISOTP
  14. Software Framework mbe.py
  15. How To: Raspberry Pi and Software Setup
  16. Where Next