Adding production serial numbers in Microchip’s 8-bit PIC

One of the Devil’s little details in designing hardware for the internet of things is the requirement that your “things” be unique.  This broadly translates into making sure that your hardware has some sort of serial number (akin to a MAC address.)  While this sounds relatively straightforward, the process itself is quite tricky when you starting thinking about manufacturing ten’s of thousands of such devices in a production environment.

The project that inspired this post is based on one of Microchip’s 8-bit microcontrollers, a 16F1517.  One of the really nice features in most of Microchip’s products is the ability to separately program a “User ID” into a predefined section of Flash memory.  This means you don’t have to re-compile or modify a hardcoded ID into unique firmware files for every part coming off the line.  Microchip also offers a service to ship parts to you pre-programmed with your ID’s!  However, there is a volume limitation and a lead-time to think about.  

My particular scenario looks like this.  I am getting thousands of boards on a regular supply from a PCB manufacturer.  The boards show up unprogrammed, and need to be flashed with firmware and a unique ID.  To do this, we’re going to make use of Microchip’s integrated production environment (IPE), the SQTP file, and some Python scripts.

The IPE is an interface between a desktop PC and a programming tool.  I’m using a Mac with an ICD3, but any combination is fine.  For volume production, the ICD3 is probably the best option.  The IPE comes bundled as part os MPLAB X here:

The SQTP file format is here for reference:

SQTP is basically a HEX file where each line of the file contains an ID for another device.  When you feed an SQTP file into the IPE, it will pull one line out of the file for every device that you program.  This guarantees that no two devices get the same ID.  The IPE also has a built in utility for generating SQTP files.  You can make them sequential, random, pseudorandom, etc.  So, one option for my scenario is to just generate one SQTP file with a bunch (thousands) of ID’s in it and then use that for programming all of my devices.  The problem with this approach is that the SQTP file itself needs to be managed…. VERY carefully.  It needs to be backed up, but also preserved every time a serial is pulled from it.  There is no room for having errors where two devices get the same ID.  What’s worse is how complicated this becomes when you have multiple stations programming boards at the same time.  Who keeps the SQTP file?  Do we generate multiple non-overlapping files?  Who manages generating those?

The approach I decided to take circumvents the SQTP file by pulling a single, unique serial number from a server on the factory floor.  Any workstation can pull a serial and be guaranteed that it’s unique since the serials themselves are generated centrally and stored in a flat database file.  One the serial number is in hand, we’ll use a “template SQTP” file and inject the new serial with the help of some Python scripting.  Finally, we’ll call the IPE stand alone JAR file directly from the Python script the program the device without any GUI.  Neat!

Step 1: Get the Python package for reading and writing Intel Hex files here:

This neat little package can read and write hex files with ease.  One issue however, is that the SQTP file write to the same address in memory over and over, since it is a slight hack on the Hex format (each line is one device, not one memory location).  This requires that we read in the HEX file in binary format instead of HEX format.  

For those interested, the Intel Hex file format is here:

Step 2: Hack apart the SQTP file and inject our serial number.

The first concern we have with our SQTP file is the address location of the serial number in the chip.  If you are putting your serials into program memory, then you can simply insert the address into each line of the HEX file in the address field.  (Warning:  I can’t make much sense of how this is encoded, so your best bet is to generate a sample file using MPLAB IPE and just use whatever it generates for the address.)

Since I want my serial number loaded into the UserID location at 0x8000, we need to do something special.  The first line of our HEX file must indicate an extended linear address.  Again, the easiest thing to do is just copy the output from IPE.  I use HexFiend to analyze what IPE is spitting out:

The first line of my hex file looks like this:


Every line thereafter is broken down very simply as follows:

: 08 0000 00 WW34XX34YY34ZZ34 CS .

Where WW, XX, YY, ZZ are the 8 hex digits of my serial number and CS is the checksum.

The last line of the hex file just closes things out like this:


As mentioned before, we’re just going to use a template SQTP file generated for a single serial number from MPLAB IPE.  This template file is passed into a Python script that reads it in, modifies it with a new serial number, and writes it out for programming.  The script even invokes the programmer from the command line and flashes the chip.

Here we go:

import os, sys, inspect
#we want to import the IntelHex library without installing it. So just copy the package into your working directory.
cmd_subfolder = os.path.realpath(os.path.abspath(os.path.join(os.path.split(inspect.getfile( inspect.currentframe() ))[0],”intelHex-1.5″)))
if cmd_subfolder not in sys.path:
sys.path.insert(0, cmd_subfolder)
from intelhex import IntelHex

newMacAddress = ‘DEADBEEF’ #this should be 8 hex digits to fit into the USER ID space at 0x8000.

pathToProgrammer = “/Applications/microchip/mplabx/”
firmwareFile = “MyFirmware.hex”

sqtp = IntelHex() #initialize the HEX reader/writer
sqtp.loadbin(“TemplateSQTP.num”) #load up the Template generated in MPLAB IPE

#we know in our template that the serial location is offset at a specific point (25)
sqtp[25] = ord(newMacAddress[0])
sqtp[26] = ord(newMacAddress[1])
sqtp[29] = ord(newMacAddress[2])
sqtp[30] = ord(newMacAddress[3])
sqtp[33] = ord(newMacAddress[4])
sqtp[34] = ord(newMacAddress[5])
sqtp[37] = ord(newMacAddress[6])
sqtp[38] = ord(newMacAddress[7])

#the serial is in place! now we need to calculate the checksum for the modified line
#first, sum up all of the ascii numbers in the line we modified
checkSum = 0
for index in range(17,40,2):
hexString = “0x” + chr(sqtp[index]) + chr(sqtp[index + 1])
checkSum += int(hexString,0)
#now, we need to take the LSB of the checksum, invert it, and mask again
checkSum = -(checkSum & 0xFF)
checkSum = (checkSum & 0xFF)
checkSumString = format(checkSum,’x’)
#now stick the new checksum into the sqtp file
sqtp[41] = ord(checkSumString[0])
sqtp[42] = ord(checkSumString[1])

#at this point, we have a SQTP ready for programming. Let’s dump it out to a one time use file.

#now, let’s flash the device by calling the ipecmd.jar directly.
#-F is the firmware file
#-M means program all memory
#-OD means power the device before programming it (Vdd first)
#-OV sets our voltage to 3.3. (Yes, this is a 5V device, but for some reason 3.3 works more reliably!)
#-P16f1517 is the PIC we are programming
#-S is the SQTP file we generated above
#-T is the tool we are using (ICD3)
#-W means that the target should be powered by the programmer

os.system(“java -jar ” + pathToProgrammer + “ipecmd.jar -F” + firmwareFile + ” -M -OD -OV3.3 -P16f1517 -SfinalSQTP.num -TPICD3 -W”)

That’s it….. I’ll post the files here as well for reference.  Enjoy!