Programming

Hello ppl,
I'm working on a windows program simulating an existing Unix program,
the program is a firmware downloader, it sends a binary firmware file through the serial port (rs232) to a Set-Top Box running under unix system too, as i'm a poor unix user and not a unix programmer i choosed the hardware way to see how the program sends the file, and i found the next :

(i analysed the communication in hexadecimal form - low level and i will write lines that i didn't understood as hexadecimal, and commas are not part of data)

1- the Unix PC sends a sort of password and awaits a specefic answer from the STB:
0x18,0x67,0x00,0x16,$,"the file name",$,00,86,9A,0C

2-the sending process starts :
a-the PC sends this line : 0x18,0x61,0x00,0x04,0x00,0x01,0xFD,0x22
b-this part is recursive and is the essential protocol's way to send the file:

I-The PC send this line of 12 bytes: 0x17,0xC0,0x1D,0xC3,0x18,0x64,0x04,0x04,0x00,0x00,0x00,0x00
where (what i understood): the first 4 bytes are some checksum, the second are a command (because they are repetitively tge same) and the third 4-bytes refer to the actual position of the file you will see by the next how it incriments.

II-The STB answers with line of 12 bytes : 0x18,0x72,0x00,0x04,0x00,0x00,0x00,0x00,0x45,0x30,0x44,0xB2
same thing, the first 4-bytes is a command (maybe) the second 4 are the loaded size (you will see the incrimentation) and the 3rd is a checksum,

III- The PC sends a block of 1024 bytes of the file
So i think that those 3 steps are : I- the pc send current position of file, II- the card responds with currently loaded size or asks for data and III- the PC sends a block of 1 kbyte data from the file

IV- The PC sends again a line of actual position 0x7F,0xEE,0xB7,0x24,0x18,0x64,0x04,0x04,0x00,0x00,0x04,0x00

V- the STB responds with 0x18, 0x72 ,0x00 ,0x04 ,0x00 ,0x00 ,0x04 ,0x00 ,0x21 ,0x5C ,0x81 ,0xB6

VI- The PC sends the second 1 kb of data

and so on till the file transmission is finished,

why i think that its a checksum? : because i tried the process with 2 different firmwares, if i repeat the operation for the same firmware many times, i find the same commands/headers/checksums in the communication,
and if i change to other firmware i find different checksums, only the incremented blocks are the same.

samples of PC commands (step I- or IV-)

Code:

		s0[12]={0x17,0xC0,0x1D,0xC3,0x18,0x64,0x04,0x04,0x00,0x00,0x00,0x00},
		s1[12]={0x7F,0xEE,0xB7,0x24,0x18,0x64,0x04,0x04,0x00,0x00,0x04,0x00},
		s2[12]={0xBB,0xFF,0xED,0xF0,0x18,0x64,0x04,0x04,0x00,0x00,0x08,0x00},
		s3[12]={0xB3,0x10,0x2C,0x9B,0x18,0x64,0x04,0x04,0x00,0x00,0x0C,0x00},
		s4[12]={0x60,0xD1,0xF3,0xE7,0x18,0x64,0x04,0x04,0x00,0x00,0x10,0x00},
		s5[12]={0x01,0x72,0x41,0x18,0x18,0x64,0x04,0x04,0x00,0x00,0x14,0x00},...

samples of STB commands:

Code:

		t0[12]={0x18, 0x72, 0x00, 0x04, 0x00, 0x00, 0x00, 0x00, 0x45, 0x30, 0x44, 0xB2},
		t1[12]={0x18, 0x72 ,0x00 ,0x04 ,0x00 ,0x00 ,0x04 ,0x00 ,0x21 ,0x5C ,0x81 ,0xB6},
		t2[12]={0x18 ,0x72 ,0x00 ,0x04 ,0x00 ,0x00 ,0x08 ,0x00 ,0x8D ,0xE9 ,0xCE ,0xBA};...

PROBLEM :
I tried sending the file with a windows application that i'm developing as the original application do, i failed: Login is ok, sending the first kbyte with loaded size response from the STB ok, second kbyte : i dont get the right answer which is "t2" array, i get continuously the "t1" array so sending failed,
I also tried to letterally send the packets sent by the unix application as they are, that i got by spying hardware, and i failed too, i tried also to insert some time breaks between sending a 1 kb bloc and receiving the answer, also not sending the second bloc unless i get the answer of first one received...
I tried many possibilities but i failed,
So what i ask you to, if some one knows this protocole tell me maybe there is such a protocol used to communicate between devices using rs232 in unix so i understand how it works
As you can see, i'm analysing this program in low level, i don't know if there is a way to analyse it in high level or if already those headers/commands seems to be familiar to you please let me know
Any information could do too much for me, thanks.

It is a very binary serial protocol! Are you sure you are sending and receiving 8 bits all data no parity? I assume it is async with a 1 stop bit. Is it full duplex, and are you using a loopback cable? Sometimes if your side drops RTS then the other side loses CD and resets, because your side is supporting a switched carrier half duplex modem and their side thinks the cable is a full duplex modem.

First thanks for reply Mr DGP,
I will give you more details about the hardware and the transmission:

The link between the PC and the board is an UART link, in the board side its in TTL 3.3v so there is no RTS CTS... just a header of 4 pins, RX,TX and GND already connected, and a VCC pin used as polarization of 3.3v of external link adapter if in need.
In my case the PC side connected by serial port to an adapter that converts the rs232 RX and TX to TTL 3.3v low level, the only weak point is that the adapter gets its polarization from two pins of the rs232 i forgot which ones i guess RTS and CTS. but still not enought reason to fear becoz its already operational in the linux PC as it is, and same that i used the link with the windows PC for TESTMODE purpose (another project) and it worked fine.

I cant get enought informations from the manufacturer but i think its a full duplex mode coz it responds to the minimum of full duplex needs ( RX TX and GND) and there is a part of the spying that shows that the communication can be full:

And yes you though right, its 8 bits no parity with 1 stop bit, with a checksum (as i assumed in the headers that i talked about) i guess parity is not needed if we already check data well reception by that checksum, and by the way it gives the right checksum for the first kbyte of data as i said.

Yes, the original tty was current loop, so send data and receive data are enough. Additional signalling is by break and long break, which are big zero's, as the line is left marking at the one level = current flowing. The 20 ma off hook current of the common POTS telephone is a tty relic!

Polarization is never an issue for ttl or rs232! RS232 +3-15 is zero, -3-15 is one. The idle data line marks at a one just like a long stop bit.

Parity costs 12.5%, is weak, so it need some checksum, but checksum can be sufficient alone. I think you said the cksum cost is 4/1024 or 1/256 or <.4%, so it's a winner, never mind the hassles of making 8 bit load data from a 7 bit payload.

So, it seems to be in the message layer. When you run good cases, does the checksum or anything change on duplicate runs? Is the checksum recognizable, some simple end around shift and half add? Some checksums shift more, some do not end around but when the upper byte is loaded makes some adjustment to prevent overflow on the next shift and half add (or add), like subtracting a right shifted version of the current sum from the sum.

Who manufactures the set top boxes? Do we have model numbers?

For polarization you havent got my point, for the PC its a rs232, but in the board side its not, its a TTL level UART, and not a rs232.

As i told you, i'm working on replacing an existing application, using that one on the same STB and with the same firmware version gives the same communication frames, the same checksums and everything is the same everytime i repeat the process on the same conditions, so those are static just depending on the firm binary file to calculate its size and checksums that i listed some of,
what i've done is spying the link between the linux PC and the board with a simple sniffering home made cable and a spying home developed software, i've got that communication, analysed it as i shown first, and tried to reconstruct that communication in a windows application,
in this application i used the same board and same binary file and used the same headers and expected the same responses, what i got is that it works till the first kbyte is sent, it gives me the right header as a response from the board
and after that (by sending the rest of file) the board responds 5 times with t1 and then quit downloading mode to show "booting the kernel..." so it failed to receive the second kbyte

what you will notice is that the kernel is ok and that the firmware is already installed before, yes it is, and same in this case when i download the firmware on it again and again using the linux PC it succeeds and shows the same checksums,
i analysed well what i got in the communication but i think that there is something special in the used protocol, maybe timing or i dont know what exactly thats why i wanted to know more about file transfer protocols,
i found a protocol too much similar to what i analysed called MODBUS but still not the same so i think its not a usual protocol.

The STB manufacturer is the company i'm doing my training with, but the developers are from other company and are not responding at all...

So, it works in LINUX but not windows? No carriage returns slipping in there?

What is the LINUX code written in?

The windows code?

Who wrote the STB firmware that carries on this conversation? I guess it is not what is being overwritten, but a sort of BIOS for the STB.

True, i realised that this file is not the real firmware file, its the first part of downloading firmware and i was told that this part (this file) purpose is to program a microcontroller in the board to make it able to connect through ethernet to continue downloading the second file of firmware with TFTP, but i dont beleive this, because the kernel is already existing in (i think) a kind of microcontroller embedded as a microOS.
I see the content of the first file as : a first part of it is unreadable, make me remember HEX files translated from a microchip PIC program so it could be a program for the second microcontroller of the STB, the second part is readable and i guess it gives how the STB will treat the second file, this is a sample of the communication in this level:




If i knew, i could do better understood this protocol, i only have the linux executable application and not a code, i asked before if there is somehow to analyse a code from an executable application, a kind of reverse engineering, found nothing.

this is what i'm working on using Labwindows/cvi, i'm just simulating this protocol without success till the moment coz of something strange in this communication.

---------- Post updated at 06:13 AM ---------- Previous update was at 03:34 AM ----------

Problem solved.
I thought that the linux PC is the master of this communication so that the response from the STB is just a report, as i didnt get what kind of protocols it is, and didnt found out how the cheks are calculated i ignored the response and sent the file continuously block by block, but when i thinked what if the STB was the master and it is probably the master as it sends the file name first as a request, so waiting the STB response before continuing transfer process is maybe needed.
So as i know that the STB will certenly respond with the known answers (t0,t1,...,t128) i didnt controlled them but just inserted a delay of 0.1 s between blocks to let the STB send the request and then be able to receive the next block.

It was my bad.
But the source of problem was that the spying link that i made is not in the middle, it was in the PC rs232 side , so i left 2 meters of the existing link on the STB side, so the communication that i got contain a retard in the STB frames :



so as you can see the 150 bytes of data made me think that the response for header A is just a report and that the STB can still receive data continuously without a breack to calculate the checks and give the response.

Now i have two options :

1- i copy parts of the communication ( file blocks and headers) and use them as they are in my application ( the stupid way of emulation)

2- search more for this strange protocol to find out how it calculates checksums and make my application autonomous.

all the headers contain 3 parts [command of 4bytes]+[file position / size loaded ] + [checksum of 4bytes]
i can let the command part as it is, the position or the size could be just incremented,
what do you think about the checksums? a 4 bytes checksum that is calculated from something that could be a block of 1024 bytes or a serie of blocks ?

I found the next: the checksum given by the STB is not a checksum, its a static answer, i tried the download with two versions of this file, one with 127 kb and one with 126 kb and i found that those are the same unless that the 127 have one more and always the last one does not contain a static checksum but the contains the number of bytes loaded in hexadecimal for exemple for the 127k file the last is 0x00 0x01 0xFD 0x22 = 130338 which is the real number of bytes contained in the file, so the response checksum doesnt make any trouble. i guess its a confirmation static code.

but for the checksum sent by the PC i didnt found any clue,