Page 2 of 3

Re: parsing of variable lenght packets with sub elements

Posted: 26 Jun 2009, 01:36
by kLabMouse
Nice!

Re: parsing of variable lenght packets with sub elements

Posted: 05 Jul 2009, 12:36
by Technology
Some stuff that we can integrate in the parsers:
- missing received packets in kore can be logged
- missing sent packets in kore can be logged using xkore (kali)

The user can sent in that log so the devs can add support.

Re: parsing of variable lenght packets with sub elements

Posted: 29 Jul 2009, 16:02
by Technology
Here is an example of the complex packet parsing we need.
In this example we parse an auction packet and a hotkey packet.

Required modules:
- Convert::Binary::C
- Data::Dumper (just to show how the perl datatype will look)

Perl script code:

Code: Select all

#!/usr/bin/env perl
package main;
use strict;
use Convert::Binary::C;
use Data::Dumper;

# Z <-> char
# C <-> unsigned char
# v <-> unsigned short
# V <-> unsigned long

#---------------------------------------------
# Create a new object and parse embedded code
#---------------------------------------------
my $c = Convert::Binary::C->new->parse(<<ENDC);

struct test1 {
	unsigned char test1, test2;
	unsigned short size;
	unsigned long pages, amount;
	struct auction {
		unsigned long ID;
		char seller[24];
		unsigned short item, type, unknown, amount;
		unsigned char identify, attribute, refine;
		unsigned short card[4];
		unsigned long price, buynow;
		char buyer[24];
		unsigned long timestamp;
	} auction[amount];
};

struct test2 {
	unsigned char test1, test2;
	struct hotkey {
		unsigned char type;
		unsigned long ID;
		unsigned short lvl;
	} hotkey[];
};

ENDC

# configuring the Convert::Binary::C object to unpack with little endian byteorder
$c->configure(ByteOrder => 'LittleEndian');

my $packed1 = pack('C*',
0x52, 0x02, 0x05, 0x01, 0x01, 0x00, 0x00, 0x00, 
0x03, 0x00, 0x00, 0x00, 0x12, 0x00, 0x00, 0x00, 
0x48, 0x79, 0x70, 0x65, 0x72, 0x69, 0x6f, 0x6e, 
0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 
0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 
0xb1, 0x04, 0x04, 0x00, 0x00, 0x00, 0x01, 0x00, 
0x01, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 
0x00, 0x00, 0x00, 0x39, 0x05, 0x00, 0x00, 0x3a, 
0x05, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 
0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 
0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 
0x00, 0x00, 0x00, 0xa7, 0x8f, 0x6d, 0x4a, 0x13, 
0x00, 0x00, 0x00, 0x48, 0x79, 0x70, 0x65, 0x72, 
0x69, 0x6f, 0x6e, 0x00, 0x00, 0x00, 0x00, 0x00, 
0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 
0x00, 0x00, 0x00, 0xfd, 0x08, 0x05, 0x00, 0x00, 
0x00, 0x01, 0x00, 0x01, 0x00, 0x00, 0x00, 0x00, 
0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0xd2, 0x04, 
0x00, 0x00, 0x2e, 0x16, 0x00, 0x00, 0x00, 0x00, 
0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 
0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 
0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0xc8, 0x8f, 
0x6d, 0x4a, 0x14, 0x00, 0x00, 0x00, 0x48, 0x79, 
0x70, 0x65, 0x72, 0x69, 0x6f, 0x6e, 0x00, 0x00, 
0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 
0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0xb1, 0x04, 
0x04, 0x00, 0x00, 0x00, 0x01, 0x00, 0x01, 0x00, 
0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 
0x00, 0xd2, 0x04, 0x00, 0x00, 0xd7, 0x11, 0x00, 
0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 
0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 
0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 
0x00, 0xe8, 0x8f, 0x6d, 0x4a);

my $packed2 = pack('H*', 'b902011c0000000a0000590200000000011a000000000001490000000a00014d0000000000014b000000000001220000000a00011d0000000900014a0000000000010f200000000001112000000000012a20000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000018e0000000000');

# configuring the Convert::Binary::C object to unpack char name[len] as string instead of array
$c->tag('test1.auction[0].seller', Format => 'String');
$c->tag('test1.auction[0].buyer', Format => 'String');

# unpacking from bytestream to perl hash using C struct
my $unpacked1 = $c->unpack('test1', $packed1);
print Dumper($unpacked1);

my $unpacked2 = $c->unpack('test2', $packed2);
print Dumper($unpacked2);

# re-packing from perl hash to bytestream using C struct
my $packed3 = $c->pack('test1', $unpacked1);
print unpack('H*', $packed3) . "\n\n";

my $packed4 = $c->pack('test2', $unpacked2);
print unpack('H*', $packed4) . "\n\n";
Console output:

Code: Select all

$VAR1 = {
          'auction' => [
                         {
                           'seller' => 'Hyperion',
                           'ID' => 18,
                           'item' => 1201,
                           'buynow' => 1338,
                           'attribute' => 0,
                           'card' => [
                                       0,
                                       0,
                                       0,
                                       0
                                     ],
                           'amount' => 1,
                           'timestamp' => 1248694183,
                           'identify' => 1,
                           'unknown' => 0,
                           'refine' => 0,
                           'buyer' => '',
                           'price' => 1337,
                           'type' => 4
                         },
                         {
                           'seller' => 'Hyperion',
                           'ID' => 19,
                           'item' => 2301,
                           'buynow' => 5678,
                           'attribute' => 0,
                           'card' => [
                                       0,
                                       0,
                                       0,
                                       0
                                     ],
                           'amount' => 1,
                           'timestamp' => 1248694216,
                           'identify' => 1,
                           'unknown' => 0,
                           'refine' => 0,
                           'buyer' => '',
                           'price' => 1234,
                           'type' => 5
                         },
                         {
                           'seller' => 'Hyperion',
                           'ID' => 20,
                           'item' => 1201,
                           'buynow' => 4567,
                           'attribute' => 0,
                           'card' => [
                                       0,
                                       0,
                                       0,
                                       0
                                     ],
                           'amount' => 1,
                           'timestamp' => 1248694248,
                           'identify' => 1,
                           'unknown' => 0,
                           'refine' => 0,
                           'buyer' => '',
                           'price' => 1234,
                           'type' => 4
                         }
                       ],
          'amount' => 3,
          'test1' => 82,
          'pages' => 1,
          'size' => 261,
          'test2' => 2
        };
$VAR1 = {
          'hotkey' => [
                        {
                          'ID' => 28,
                          'lvl' => 10,
                          'type' => 1
                        },
                        {
                          'ID' => 601,
                          'lvl' => 0,
                          'type' => 0
                        },
                        {
                          'ID' => 26,
                          'lvl' => 0,
                          'type' => 1
                        },
                        {
                          'ID' => 73,
                          'lvl' => 10,
                          'type' => 1
                        },
                        {
                          'ID' => 77,
                          'lvl' => 0,
                          'type' => 1
                        },
                        {
                          'ID' => 75,
                          'lvl' => 0,
                          'type' => 1
                        },
                        {
                          'ID' => 34,
                          'lvl' => 10,
                          'type' => 1
                        },
                        {
                          'ID' => 29,
                          'lvl' => 9,
                          'type' => 1
                        },
                        {
                          'ID' => 74,
                          'lvl' => 0,
                          'type' => 1
                        },
                        {
                          'ID' => 8207,
                          'lvl' => 0,
                          'type' => 1
                        },
                        {
                          'ID' => 8209,
                          'lvl' => 0,
                          'type' => 1
                        },
                        {
                          'ID' => 8234,
                          'lvl' => 0,
                          'type' => 1
                        },
                        {
                          'ID' => 0,
                          'lvl' => 0,
                          'type' => 0
                        },
                        {
                          'ID' => 0,
                          'lvl' => 0,
                          'type' => 0
                        },
                        {
                          'ID' => 0,
                          'lvl' => 0,
                          'type' => 0
                        },
                        {
                          'ID' => 0,
                          'lvl' => 0,
                          'type' => 0
                        },
                        {
                          'ID' => 0,
                          'lvl' => 0,
                          'type' => 0
                        },
                        {
                          'ID' => 0,
                          'lvl' => 0,
                          'type' => 0
                        },
                        {
                          'ID' => 0,
                          'lvl' => 0,
                          'type' => 0
                        },
                        {
                          'ID' => 0,
                          'lvl' => 0,
                          'type' => 0
                        },
                        {
                          'ID' => 0,
                          'lvl' => 0,
                          'type' => 0
                        },
                        {
                          'ID' => 0,
                          'lvl' => 0,
                          'type' => 0
                        },
                        {
                          'ID' => 0,
                          'lvl' => 0,
                          'type' => 0
                        },
                        {
                          'ID' => 0,
                          'lvl' => 0,
                          'type' => 0
                        },
                        {
                          'ID' => 0,
                          'lvl' => 0,
                          'type' => 0
                        },
                        {
                          'ID' => 0,
                          'lvl' => 0,
                          'type' => 0
                        },
                        {
                          'ID' => 142,
                          'lvl' => 0,
                          'type' => 1
                        }
                      ],
          'test1' => 185,
          'test2' => 2
        };
520205010100000003000000120000004879706572696f6e00000000000000000000000000000000
b1040400000001000100000000000000000000390500003a05000000000000000000000000000000
0000000000000000000000a78f6d4a130000004879706572696f6e00000000000000000000000000
000000fd080500000001000100000000000000000000d20400002e16000000000000000000000000
0000000000000000000000000000c88f6d4a140000004879706572696f6e00000000000000000000
000000000000b1040400000001000100000000000000000000d2040000d711000000000000000000
0000000000000000000000000000000000e88f6d4a

b902011c0000000a0000590200000000011a000000000001490000000a00014d0000000000014b00
0000000001220000000a00011d0000000900014a0000000000010f20000000000111200000000001
2a200000000000000000000000000000000000000000000000000000000000000000000000000000
00000000000000000000000000000000000000000000000000000000000000000000000000000000
000000000000000000000000000000000000000000000000018e0000000000

Re: parsing of variable lenght packets with sub elements

Posted: 29 Jul 2009, 17:01
by kLabMouse
Technology
Nice!!!
Need just one addition, the eval for array elements number.

Re: parsing of variable lenght packets with sub elements

Posted: 29 Jul 2009, 17:43
by Technology
kLabMouse wrote:Technology
Nice!!!
Need just one addition, the eval for array elements number.
Can you explain what you mean exactly or by the hand of an example?

Re: parsing of variable lenght packets with sub elements

Posted: 30 Jul 2009, 03:44
by kLabMouse
Some packet's can have a structure:

Code: Select all

short param_1
short param_2
struct {
  something
} array[(packet_len-2-4-2)/struct_size]
short param_3
You see the "array" ??? Well, number of elements is calculated from packet len, and there is additional param_3 at the end of packet.
So it need to be parsed good.

Re: parsing of variable lenght packets with sub elements

Posted: 15 Dec 2009, 20:18
by Technology
I've taken a look at the structs and all inner structs/datatypes that have a [...] seem to have it at the end of the packet.

example:

Code: Select all

// packet 0x155
struct PACKET_CZ_REQ_CHANGE_MEMBERPOS {
  /* this+0x0 */ short PacketType
  /* this+0x2 */ short PacketLength
  struct MEMBER_POSITION_INFO memberInfo[...] {
    /* this+0x0 */ int AID;
    /* this+0x4 */ int GID;
    /* this+0x8 */ int positionID;
  }
}
or also:

Code: Select all

// packet 0x155
struct PACKET_CZ_REQ_CHANGE_MEMBERPOS {
	short PacketType;
	short PacketLength;
	struct MEMBER_POSITION_INFO {
		int AID;
		int GID;
		int positionID;
	} memberInfo[];
};

btw i think its possible to use this if you need to calculate lengths of struct arrays:
The final and most powerful way to define a Dimension tag is to pass it a subroutine reference. The referenced subroutine can execute whatever code is neccessary to determine the size of the tagged array:

Code: Select all

 sub get_size
  {
    my $m = shift;
    return $m->{hdr}{len}[0] / $m->{hdr}{len}[1];
  }
  
  $c->tag('more_complex.data', Dimension => \&get_size);
  
  $u = $c->unpack('more_complex', $data);
As you can guess from the above code, the subroutine is being passed a reference to hash that stores the already unpacked part of the compound embedding the tagged array. This is the result:

Code: Select all

  $u = {
    'hdr' => {
      'len' => [
        42,
        7
      ]
    },
    'data' => [
      1,
      2,
      3,
      4,
      5,
      6
    ]
  };
to do a bytesToString, we can use something like this:

Code: Select all

$c->tag('ProtoId', Hooks => { pack   => \&ProtoId_pack,
                                unpack => \&ProtoId_unpack });

Re: parsing of variable lenght packets with sub elements

Posted: 19 Jan 2010, 07:33
by kLabMouse
Nice. Should We implement it? or only in 3.0 ?

Re: parsing of variable lenght packets with sub elements

Posted: 19 Jan 2010, 09:24
by kali
Yeah looks really nice.

I guess once we release the next stable version, we can start doing this? It doesn't have to be 3.0 could be 2.1.1 or 2.2 or something.

Re: parsing of variable lenght packets with sub elements

Posted: 19 Jan 2010, 09:44
by kLabMouse
kali wrote:It doesn't have to be 3.0 could be 2.1.1 or 2.2 or something.
Yep. But it's a big Update. so better 2.2 or even 3.0