parsing of variable lenght packets with sub elements

Wrote new code? Fixed a bug? Want to discuss technical stuff? Feel free to post it here.

Moderator: Moderators

Message
Author

Technology
Super Moderators
Super Moderators
Posts: 801
Joined: 06 May 2008, 12:47
Noob?: No

Re: parsing of variable lenght packets with sub elements

#12 Post by Technology »

Some stuff that we can integrate in the parsers:
- missing received packets in kore can be logged
- missing sent packets in kore can be logged using xkore (kali)

The user can sent in that log so the devs can add support.
One ST0 to rule them all? One PE viewer to find them!
One ST_kRO to bring them all and in the darkness bind them...

Mount Doom awaits us, fellowship of OpenKore!

Technology
Super Moderators
Super Moderators
Posts: 801
Joined: 06 May 2008, 12:47
Noob?: No

Re: parsing of variable lenght packets with sub elements

#13 Post by Technology »

Here is an example of the complex packet parsing we need.
In this example we parse an auction packet and a hotkey packet.

Required modules:
- Convert::Binary::C
- Data::Dumper (just to show how the perl datatype will look)

Perl script code:

Code: Select all

#!/usr/bin/env perl
package main;
use strict;
use Convert::Binary::C;
use Data::Dumper;

# Z <-> char
# C <-> unsigned char
# v <-> unsigned short
# V <-> unsigned long

#---------------------------------------------
# Create a new object and parse embedded code
#---------------------------------------------
my $c = Convert::Binary::C->new->parse(<<ENDC);

struct test1 {
	unsigned char test1, test2;
	unsigned short size;
	unsigned long pages, amount;
	struct auction {
		unsigned long ID;
		char seller[24];
		unsigned short item, type, unknown, amount;
		unsigned char identify, attribute, refine;
		unsigned short card[4];
		unsigned long price, buynow;
		char buyer[24];
		unsigned long timestamp;
	} auction[amount];
};

struct test2 {
	unsigned char test1, test2;
	struct hotkey {
		unsigned char type;
		unsigned long ID;
		unsigned short lvl;
	} hotkey[];
};

ENDC

# configuring the Convert::Binary::C object to unpack with little endian byteorder
$c->configure(ByteOrder => 'LittleEndian');

my $packed1 = pack('C*',
0x52, 0x02, 0x05, 0x01, 0x01, 0x00, 0x00, 0x00, 
0x03, 0x00, 0x00, 0x00, 0x12, 0x00, 0x00, 0x00, 
0x48, 0x79, 0x70, 0x65, 0x72, 0x69, 0x6f, 0x6e, 
0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 
0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 
0xb1, 0x04, 0x04, 0x00, 0x00, 0x00, 0x01, 0x00, 
0x01, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 
0x00, 0x00, 0x00, 0x39, 0x05, 0x00, 0x00, 0x3a, 
0x05, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 
0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 
0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 
0x00, 0x00, 0x00, 0xa7, 0x8f, 0x6d, 0x4a, 0x13, 
0x00, 0x00, 0x00, 0x48, 0x79, 0x70, 0x65, 0x72, 
0x69, 0x6f, 0x6e, 0x00, 0x00, 0x00, 0x00, 0x00, 
0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 
0x00, 0x00, 0x00, 0xfd, 0x08, 0x05, 0x00, 0x00, 
0x00, 0x01, 0x00, 0x01, 0x00, 0x00, 0x00, 0x00, 
0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0xd2, 0x04, 
0x00, 0x00, 0x2e, 0x16, 0x00, 0x00, 0x00, 0x00, 
0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 
0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 
0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0xc8, 0x8f, 
0x6d, 0x4a, 0x14, 0x00, 0x00, 0x00, 0x48, 0x79, 
0x70, 0x65, 0x72, 0x69, 0x6f, 0x6e, 0x00, 0x00, 
0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 
0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0xb1, 0x04, 
0x04, 0x00, 0x00, 0x00, 0x01, 0x00, 0x01, 0x00, 
0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 
0x00, 0xd2, 0x04, 0x00, 0x00, 0xd7, 0x11, 0x00, 
0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 
0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 
0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 
0x00, 0xe8, 0x8f, 0x6d, 0x4a);

my $packed2 = pack('H*', 'b902011c0000000a0000590200000000011a000000000001490000000a00014d0000000000014b000000000001220000000a00011d0000000900014a0000000000010f200000000001112000000000012a20000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000018e0000000000');

# configuring the Convert::Binary::C object to unpack char name[len] as string instead of array
$c->tag('test1.auction[0].seller', Format => 'String');
$c->tag('test1.auction[0].buyer', Format => 'String');

# unpacking from bytestream to perl hash using C struct
my $unpacked1 = $c->unpack('test1', $packed1);
print Dumper($unpacked1);

my $unpacked2 = $c->unpack('test2', $packed2);
print Dumper($unpacked2);

# re-packing from perl hash to bytestream using C struct
my $packed3 = $c->pack('test1', $unpacked1);
print unpack('H*', $packed3) . "\n\n";

my $packed4 = $c->pack('test2', $unpacked2);
print unpack('H*', $packed4) . "\n\n";
Console output:

Code: Select all

$VAR1 = {
          'auction' => [
                         {
                           'seller' => 'Hyperion',
                           'ID' => 18,
                           'item' => 1201,
                           'buynow' => 1338,
                           'attribute' => 0,
                           'card' => [
                                       0,
                                       0,
                                       0,
                                       0
                                     ],
                           'amount' => 1,
                           'timestamp' => 1248694183,
                           'identify' => 1,
                           'unknown' => 0,
                           'refine' => 0,
                           'buyer' => '',
                           'price' => 1337,
                           'type' => 4
                         },
                         {
                           'seller' => 'Hyperion',
                           'ID' => 19,
                           'item' => 2301,
                           'buynow' => 5678,
                           'attribute' => 0,
                           'card' => [
                                       0,
                                       0,
                                       0,
                                       0
                                     ],
                           'amount' => 1,
                           'timestamp' => 1248694216,
                           'identify' => 1,
                           'unknown' => 0,
                           'refine' => 0,
                           'buyer' => '',
                           'price' => 1234,
                           'type' => 5
                         },
                         {
                           'seller' => 'Hyperion',
                           'ID' => 20,
                           'item' => 1201,
                           'buynow' => 4567,
                           'attribute' => 0,
                           'card' => [
                                       0,
                                       0,
                                       0,
                                       0
                                     ],
                           'amount' => 1,
                           'timestamp' => 1248694248,
                           'identify' => 1,
                           'unknown' => 0,
                           'refine' => 0,
                           'buyer' => '',
                           'price' => 1234,
                           'type' => 4
                         }
                       ],
          'amount' => 3,
          'test1' => 82,
          'pages' => 1,
          'size' => 261,
          'test2' => 2
        };
$VAR1 = {
          'hotkey' => [
                        {
                          'ID' => 28,
                          'lvl' => 10,
                          'type' => 1
                        },
                        {
                          'ID' => 601,
                          'lvl' => 0,
                          'type' => 0
                        },
                        {
                          'ID' => 26,
                          'lvl' => 0,
                          'type' => 1
                        },
                        {
                          'ID' => 73,
                          'lvl' => 10,
                          'type' => 1
                        },
                        {
                          'ID' => 77,
                          'lvl' => 0,
                          'type' => 1
                        },
                        {
                          'ID' => 75,
                          'lvl' => 0,
                          'type' => 1
                        },
                        {
                          'ID' => 34,
                          'lvl' => 10,
                          'type' => 1
                        },
                        {
                          'ID' => 29,
                          'lvl' => 9,
                          'type' => 1
                        },
                        {
                          'ID' => 74,
                          'lvl' => 0,
                          'type' => 1
                        },
                        {
                          'ID' => 8207,
                          'lvl' => 0,
                          'type' => 1
                        },
                        {
                          'ID' => 8209,
                          'lvl' => 0,
                          'type' => 1
                        },
                        {
                          'ID' => 8234,
                          'lvl' => 0,
                          'type' => 1
                        },
                        {
                          'ID' => 0,
                          'lvl' => 0,
                          'type' => 0
                        },
                        {
                          'ID' => 0,
                          'lvl' => 0,
                          'type' => 0
                        },
                        {
                          'ID' => 0,
                          'lvl' => 0,
                          'type' => 0
                        },
                        {
                          'ID' => 0,
                          'lvl' => 0,
                          'type' => 0
                        },
                        {
                          'ID' => 0,
                          'lvl' => 0,
                          'type' => 0
                        },
                        {
                          'ID' => 0,
                          'lvl' => 0,
                          'type' => 0
                        },
                        {
                          'ID' => 0,
                          'lvl' => 0,
                          'type' => 0
                        },
                        {
                          'ID' => 0,
                          'lvl' => 0,
                          'type' => 0
                        },
                        {
                          'ID' => 0,
                          'lvl' => 0,
                          'type' => 0
                        },
                        {
                          'ID' => 0,
                          'lvl' => 0,
                          'type' => 0
                        },
                        {
                          'ID' => 0,
                          'lvl' => 0,
                          'type' => 0
                        },
                        {
                          'ID' => 0,
                          'lvl' => 0,
                          'type' => 0
                        },
                        {
                          'ID' => 0,
                          'lvl' => 0,
                          'type' => 0
                        },
                        {
                          'ID' => 0,
                          'lvl' => 0,
                          'type' => 0
                        },
                        {
                          'ID' => 142,
                          'lvl' => 0,
                          'type' => 1
                        }
                      ],
          'test1' => 185,
          'test2' => 2
        };
520205010100000003000000120000004879706572696f6e00000000000000000000000000000000
b1040400000001000100000000000000000000390500003a05000000000000000000000000000000
0000000000000000000000a78f6d4a130000004879706572696f6e00000000000000000000000000
000000fd080500000001000100000000000000000000d20400002e16000000000000000000000000
0000000000000000000000000000c88f6d4a140000004879706572696f6e00000000000000000000
000000000000b1040400000001000100000000000000000000d2040000d711000000000000000000
0000000000000000000000000000000000e88f6d4a

b902011c0000000a0000590200000000011a000000000001490000000a00014d0000000000014b00
0000000001220000000a00011d0000000900014a0000000000010f20000000000111200000000001
2a200000000000000000000000000000000000000000000000000000000000000000000000000000
00000000000000000000000000000000000000000000000000000000000000000000000000000000
000000000000000000000000000000000000000000000000018e0000000000
One ST0 to rule them all? One PE viewer to find them!
One ST_kRO to bring them all and in the darkness bind them...

Mount Doom awaits us, fellowship of OpenKore!

User avatar
kLabMouse
Administrator
Administrator
Posts: 1301
Joined: 24 Apr 2008, 12:02

Re: parsing of variable lenght packets with sub elements

#14 Post by kLabMouse »

Technology
Nice!!!
Need just one addition, the eval for array elements number.

Technology
Super Moderators
Super Moderators
Posts: 801
Joined: 06 May 2008, 12:47
Noob?: No

Re: parsing of variable lenght packets with sub elements

#15 Post by Technology »

kLabMouse wrote:Technology
Nice!!!
Need just one addition, the eval for array elements number.
Can you explain what you mean exactly or by the hand of an example?
One ST0 to rule them all? One PE viewer to find them!
One ST_kRO to bring them all and in the darkness bind them...

Mount Doom awaits us, fellowship of OpenKore!

User avatar
kLabMouse
Administrator
Administrator
Posts: 1301
Joined: 24 Apr 2008, 12:02

Re: parsing of variable lenght packets with sub elements

#16 Post by kLabMouse »

Some packet's can have a structure:

Code: Select all

short param_1
short param_2
struct {
  something
} array[(packet_len-2-4-2)/struct_size]
short param_3
You see the "array" ??? Well, number of elements is calculated from packet len, and there is additional param_3 at the end of packet.
So it need to be parsed good.

Technology
Super Moderators
Super Moderators
Posts: 801
Joined: 06 May 2008, 12:47
Noob?: No

Re: parsing of variable lenght packets with sub elements

#17 Post by Technology »

I've taken a look at the structs and all inner structs/datatypes that have a [...] seem to have it at the end of the packet.

example:

Code: Select all

// packet 0x155
struct PACKET_CZ_REQ_CHANGE_MEMBERPOS {
  /* this+0x0 */ short PacketType
  /* this+0x2 */ short PacketLength
  struct MEMBER_POSITION_INFO memberInfo[...] {
    /* this+0x0 */ int AID;
    /* this+0x4 */ int GID;
    /* this+0x8 */ int positionID;
  }
}
or also:

Code: Select all

// packet 0x155
struct PACKET_CZ_REQ_CHANGE_MEMBERPOS {
	short PacketType;
	short PacketLength;
	struct MEMBER_POSITION_INFO {
		int AID;
		int GID;
		int positionID;
	} memberInfo[];
};

btw i think its possible to use this if you need to calculate lengths of struct arrays:
The final and most powerful way to define a Dimension tag is to pass it a subroutine reference. The referenced subroutine can execute whatever code is neccessary to determine the size of the tagged array:

Code: Select all

 sub get_size
  {
    my $m = shift;
    return $m->{hdr}{len}[0] / $m->{hdr}{len}[1];
  }
  
  $c->tag('more_complex.data', Dimension => \&get_size);
  
  $u = $c->unpack('more_complex', $data);
As you can guess from the above code, the subroutine is being passed a reference to hash that stores the already unpacked part of the compound embedding the tagged array. This is the result:

Code: Select all

  $u = {
    'hdr' => {
      'len' => [
        42,
        7
      ]
    },
    'data' => [
      1,
      2,
      3,
      4,
      5,
      6
    ]
  };
to do a bytesToString, we can use something like this:

Code: Select all

$c->tag('ProtoId', Hooks => { pack   => \&ProtoId_pack,
                                unpack => \&ProtoId_unpack });
One ST0 to rule them all? One PE viewer to find them!
One ST_kRO to bring them all and in the darkness bind them...

Mount Doom awaits us, fellowship of OpenKore!

User avatar
kLabMouse
Administrator
Administrator
Posts: 1301
Joined: 24 Apr 2008, 12:02

Re: parsing of variable lenght packets with sub elements

#18 Post by kLabMouse »

Nice. Should We implement it? or only in 3.0 ?

kali
OpenKore Monk
OpenKore Monk
Posts: 457
Joined: 04 Apr 2008, 10:10

Re: parsing of variable lenght packets with sub elements

#19 Post by kali »

Yeah looks really nice.

I guess once we release the next stable version, we can start doing this? It doesn't have to be 3.0 could be 2.1.1 or 2.2 or something.
Got your topic trashed by a mod?

Trashing topics is one click, and moving a topic to its proper forum is a lot harder. You expend the least effort in deciding where to post, mods expend the least effort by trashing.

Have a nice day.

User avatar
kLabMouse
Administrator
Administrator
Posts: 1301
Joined: 24 Apr 2008, 12:02

Re: parsing of variable lenght packets with sub elements

#20 Post by kLabMouse »

kali wrote:It doesn't have to be 3.0 could be 2.1.1 or 2.2 or something.
Yep. But it's a big Update. so better 2.2 or even 3.0

Post Reply