Skip to content

v1_1_c_overview

Jump to bottom
Takatoshi Kondo edited this page Oct 26, 2015 · 18 revisions

msgpack-c C version users guide

This document is under construction.

How to use

Include msgpack.h header file.

#include <msgpack.h>

Link msgpack library. When you use UNIX, the library name is libmsgpack.a. When you use Windows, the library name is msgpack.lib for static link and msgpack_import.lib for dynamic link.

Packing

When you pack your data, use buffers and a packer.

msgpack_sbuffer sbuf; /* buffer */
msgpack_packer pk;    /* packer */

msgpack_sbuffer_init(&sbuf); /* initialize buffer */
msgpack_packer_init(&pk, &sbuf, msgpack_sbuffer_write); /* initialize packer */

Buffers

The buffers store packed byte data. The buffers are used by packer. msgpack-c provides four buffers. sbuffer, fbuffer, zbuffer, and vrefbuffer.

sbuffer is a simple memory buffer. It uses malloc, realloc, and free. fbuffer is a file buffer. It uses FILE structure in C standard library. zbuffer is a compressing buffer using zlib. So packed data is compressed. vrefbuffer is an iovec based buffer. So packed data is stored as iovec.

Initialize, Destroy, and Write

All buffers have at least three functions. Initialize, Destroy, and Write.

Buffer Initialize Destroy Write
sbuffer msgpack_sbuffer_init msgpack_sbuffer_destroy msgpack_sbuffer_write
zbuffer msgpack_zbuffer_init msgpack_zbuffer_destroy msgpack_zbuffer_write
vrefbuffer msgpack_vrefbuffer_init msgpack_vrefbuffer_destroy msgpack_vrefbuffer_write
fbuffer fopen fclose msgpack_fbuffer_write

When you initialize packer, you pass a pointer to the buffer as the second argument, and pass write function pointer as the third argument. Packer calls the write function through the pointer internally.

The buffers have additional buffer specific functions. Please check the source code.

Packer

Packer provides packing mechanism. Packer requires a buffer. In order to use packer, call msgpack_packer_init() as follows:

msgpack_sbuffer sbuf; /* buffer */
msgpack_packer pk;    /* packer */

msgpack_sbuffer_init(&sbuf); /* initialize buffer */
msgpack_packer_init(&pk, &sbuf, msgpack_sbuffer_write); /* initialize packer */

After initializing, you can call the following packing functions:

int msgpack_pack_char(msgpack_packer* pk, char d);

int msgpack_pack_signed_char(msgpack_packer* pk, signed char d);
int msgpack_pack_short(msgpack_packer* pk, short d);
int msgpack_pack_int(msgpack_packer* pk, int d);
int msgpack_pack_long(msgpack_packer* pk, long d);
int msgpack_pack_long_long(msgpack_packer* pk, long long d);
int msgpack_pack_unsigned_char(msgpack_packer* pk, unsigned char d);
int msgpack_pack_unsigned_short(msgpack_packer* pk, unsigned short d);
int msgpack_pack_unsigned_int(msgpack_packer* pk, unsigned int d);
int msgpack_pack_unsigned_long(msgpack_packer* pk, unsigned long d);
int msgpack_pack_unsigned_long_long(msgpack_packer* pk, unsigned long long d);

int msgpack_pack_uint8(msgpack_packer* pk, uint8_t d);
int msgpack_pack_uint16(msgpack_packer* pk, uint16_t d);
int msgpack_pack_uint32(msgpack_packer* pk, uint32_t d);
int msgpack_pack_uint64(msgpack_packer* pk, uint64_t d);
int msgpack_pack_int8(msgpack_packer* pk, int8_t d);
int msgpack_pack_int16(msgpack_packer* pk, int16_t d);
int msgpack_pack_int32(msgpack_packer* pk, int32_t d);
int msgpack_pack_int64(msgpack_packer* pk, int64_t d);

int msgpack_pack_fix_uint8(msgpack_packer* pk, uint8_t d);
int msgpack_pack_fix_uint16(msgpack_packer* pk, uint16_t d);
int msgpack_pack_fix_uint32(msgpack_packer* pk, uint32_t d);
int msgpack_pack_fix_uint64(msgpack_packer* pk, uint64_t d);
int msgpack_pack_fix_int8(msgpack_packer* pk, int8_t d);
int msgpack_pack_fix_int16(msgpack_packer* pk, int16_t d);
int msgpack_pack_fix_int32(msgpack_packer* pk, int32_t d);
int msgpack_pack_fix_int64(msgpack_packer* pk, int64_t d);

int msgpack_pack_float(msgpack_packer* pk, float d);
int msgpack_pack_double(msgpack_packer* pk, double d);

int msgpack_pack_nil(msgpack_packer* pk);
int msgpack_pack_true(msgpack_packer* pk);
int msgpack_pack_false(msgpack_packer* pk);

int msgpack_pack_array(msgpack_packer* pk, size_t n);

int msgpack_pack_map(msgpack_packer* pk, size_t n);

int msgpack_pack_str(msgpack_packer* pk, size_t l);
int msgpack_pack_str_body(msgpack_packer* pk, const void* b, size_t l);

int msgpack_pack_v4raw(msgpack_packer* pk, size_t l);
int msgpack_pack_v4raw_body(msgpack_packer* pk, const void* b, size_t l);

int msgpack_pack_bin(msgpack_packer* pk, size_t l);
int msgpack_pack_bin_body(msgpack_packer* pk, const void* b, size_t l);

int msgpack_pack_ext(msgpack_packer* pk, size_t l, int8_t type);
int msgpack_pack_ext_body(msgpack_packer* pk, const void* b, size_t l);

int msgpack_pack_object(msgpack_packer* pk, msgpack_object d);

Those functions are corresponding to msgpack format

To pack STR, RAW(v4 format), BIN, and EXT, call msgpack_pack_[str|v4raw|bin|ext] to provide a size and then call msgpack_pack_[str|v4raw|bin|ext]_body to provide a payload. To pack ARRAY, call msgpack_pack_array to provide the number of elements and then call any msgpack_pack_* functions the number times. To pack MAP, call msgpack_pack_map to provide the number of elements and then call any msgpack_pack_* functions the number*2 times (key and value).

When you use packer as value (not pointer), you don't need to call destroy function. Be careful that msgpack_packer_free is not a destroy function for a packer as value. It is for msgpack_packer_new. If you want to create a packer on heap, call msgpack_packer_new to create a packer and msgpack_packer_free to destroy(free) the packer.

Unpack

Using unpacker

When you unpack message pack format data, use msgpack_unpacker. When you use msgpack_unpacker on the stack, use msgpack_unpacker_init() and msgpack_unpacker_destroy() as follows:

/* unpacker on the stack */
msgpack_unpacker unp;

/* Initialize the unpacker. 100 means initial buffer size. */
/* It can expand later. */
bool result = msgpack_unpacker_init(&unp, 100);
/* If memory allocation is failed, result is false, else result is true. */
if (result) {
    /* Do unpacking */
}
msgpack_unpacker_destroy(&unp);

When you use msgpack_unpacker on the heap, use msgpack_unpacker_new() and msgpack_unpacker_free() as follows:

/* unpacker on the stack */
msgpack_unpacker* unp = msgpack_unpacker_new(100);
if (unp) {
    /* Do unpacking */
}
msgpack_unpacker_free(unp);

Here is declaration of the functions above:

bool msgpack_unpacker_init(msgpack_unpacker* mpac, size_t initial_buffer_size);
void msgpack_unpacker_destroy(msgpack_unpacker* mpac);

msgpack_unpacker* msgpack_unpacker_new(size_t initial_buffer_size);
void msgpack_unpacker_free(msgpack_unpacker* mpac);

After initializing msgpack_unpacker, do unpacking process. I assume that unp is on the stack in the following codes.

First, check the msgpack_unpacker's buffer capacity. Let's say request_size is the size you need. You can check the buffer size and expand it if the buffer size is not enough.

    if (msgpack_unpacker_buffer_capacity(&unp) < request_size) {
        bool result = msgpack_unpacker_reserve_buffer(&unp, request_size);
        if (!result) {
            /* Memory allocation error. */
        }
    }

Now, msgpack_unpacker has enough buffer size. You can get buffer pointer using msgpack_unpacker_buffer(). And then, write the data to the buffer.

    memcpy(msgpack_unpacker_buffer(&unp), data, request_size);

The number of writing bytes is might not always equal to reqest_size. See https://github.com/msgpack/msgpack-c/blob/master/example/c/lib_buffer_unpack.c.

After writing, you need to notify the number of wrote bytes to msgpack_unpacker using msgpack_unpacker_buffer_consumed().

    msgpack_unpacker_buffer_consumed(&unp, request_size);

Now, msgpack_unpacker is ready to unpack. An unpacked object is stored in msgpack_unpacked. You need to initialize it using msgpack_unpacked_init(). Then, call msgpack_unpack_next().

    {
        msgpack_unpacked und;
        msgpack_unpacked_return ret;
        msgpack_unpacked_init(&und);
        ret = msgpack_unpack_next(&unp, &upd);
        switch(ret) {
        case MSGPACK_UNPACK_SUCCESS:
            {
                msgpack_object obj = ret.data;
                /* Extract msgpack_object and use it. */
            }
            break;
        case MSGPACK_UNPACK_EXTRA_BYTES:
            {
                msgpack_object obj = ret.data;
                /* Extract msgpack_object and use it. */

                /* call msgpack_unpack_next(&unp, &upd) again */
                /* it is usually in the loop. */
            }
            break;
        case MSGPACK_UNPACK_CONTINUE:
            /* cheking capacity, reserve buffer, copy additional data to the buffer, */
            /* notify consumed buffer size, then call msgpack_unpack_next(&unp, &upd) again */
            break;
        case MSGPACK_UNPACK_PARSE_ERROR:
            /* Error process */
            break;
        }
    }

The meaning of msgpack_unpacked_return is as follows:

return code meaning
MSGPACK_UNPACK_SUCCESS Successfully unpacked. No extra bytes.
MSGPACK_UNPACK_EXTRA_BYTES Successfully unpacked. There are extra bytes.
MSGPACK_UNPACK_CONTINUE msgpack format data is incomplete. You need to provide additional bytes.
MSGPACK_UNPACK_PARSE_ERROR msgpack format data is invalid.

If msgpack_unpacked_return is MSGPACK_UNPACK_SUCCESS or MSGPACK_UNPACK_EXTRA_BYTES, msgpack_unpacked has unpacked msgpack_object. It is set as the member variable named data.

After unpacked, you need to destroy msgpack_unpacked using msgpack_unpacked_destroy().

    msgpack_unpacked_destroy(&und);

You don't need to call msgpack_unpack_destroy() between calling msgpack_unpack_next(), because msgpack_unpack_next() calls msgpack_unpack_destroy() internally. This means the msgpack_object's life time would be over the next msgpack_unpack_next() is called. If you want to extend the lifetime of msgpack_object, you need to call msgpack_unpacker_release_zone() and manage returned msgpack_zone.

Here is the memory model of msgpack_unpacked: unpacker behavior

Using unpack function

This function is obsolete. Unfortunately, some of examples still uses the function...

msgpack_unpack_return
msgpack_unpack(const char* data, size_t len, size_t* off,
               msgpack_zone* result_zone, msgpack_object* result);

If msgpack data contains STR, BIN, or EXT, unpacked msgpack_object refer to msgpack data buffer.

Zone

Zone is a kind of memory pool. Zone is used by unpacker and msgpack_unpack() (obsolete). msgpack_object is allocated on the zone. When you use an unpacker, you don't need to care about zone except you call msgpack_unpacker_release_zone(). Zone is generated and managed by the unpacker. When you call msgpack_unpacker_release_zone(), you get a pointer of the zone as the return value. To destroy the zone, you need to call msgpack_zone_destroy() when the object allocated on the zone is no longer used.

void msgpack_zone_destroy(msgpack_zone* zone);
Clone this wiki locally