That's pretty close, but you are are ignoring the protocols. The IP address gets the packet to the box. The next thing that happens is the IP layer looks at the protocol. The protocol might be TCP, UDP, ICMP, and a few others. So the IP layer hands the packet to the proper protocol layer. This will usually be TCP or UDP. Both TCP and UDP will then look at the port number. If a program has that port open it will get the packet.

The program could be client or server.

Try this:
grep telnet /etc/services

You should see:
telnet 23/tcp

So a telnet daemon might be running and if it is, it would be using port 23 but under the tcp protocol. A packet coming in to port 23 with the udp protocol would be an error of some kind. It is even possible that some other program is using 23/udp but that would be rare.

I don't want to write a complete book here, but one more point... tcp is connection oriented. That means that your average tcp packet has a second port and a second ip address that tell where it came from. So there 4 things to look at:
destination ip address
destination port
source ip address
source ip port

All 4 have to match. That is how you can have 20 telnetd's running. They all have the same local ip address and they use port 23. But it takes all 4 numbers to define a socket.

There is more to this and you really should read a book. I suggest TCP/IP Illustrated Volume 1 by Rich Stevens.