> But when you’re working with files that only your software will access, unpacking it byte by byte will slow down IO by a huge factor compared to both mmap, and read/write of large blocks (the latter will likely translate to DMA i.e. the CPU will be free to do something else).
If:
* you're working with files that only your software will access AND
* you control, and understand, the CPU architecture of any machine that will touch that data structure and you KNOW that it will never change for the entire lifetime of that piece of data AND
* you always use the same version, or an ABI-compatible future version, of the same compiler that you KNOW will always lay out data the same way and you know what that way is AND
* you either don't need to touch this data structure in other programming languages, or you KNOW that this won't be an issue (for example because your programming language implementation was written in C, compiled against the same version of the compiler, and has an FFI that understands C structs)
THEN, you may proceed to mmap structs into memory. In practice these constraints are fairly commonplace; for example, NetBSD wscons device drivers report HID events which are specified in a struct, and because the HID events are likely never to leave the originating machine, only be passed from kernel space to user space, it makes sense to simply read(2) them straight into the struct.
And there's certainly nothing wrong with snarfing a large file into a char[], but when it comes time to extract meaning from it, unless you are absolutely sure that the underlying assumptions regarding data layout will never ever change, it will only cause marginal harm to do everything in byte offsets and shift and OR bytes to yield final values -- and this is the best portable way to access the data therein, agnostic of details about the compiler and CPU architecture.
In general it's a good idea to err on the side of caution by default, profile, and then optimize the hot paths as necessary bearing the constraints you're assuming in mind (and perhaps documenting them for good measure).
Maybe my initial statement was too strong, but I shudder whenever I see comments of the form "It's so convenient to just mmap() that sucker into a struct and access the fields!" Because they've never had to deal with the consequences of trying to access a struct from a Microsoft compiler serialized to disk, and finding out that GNU compilers have a quite different notion of how structure members are to be laid out in memory...
> In general it's a good idea to err on the side of caution by default, profile, and then optimize the hot paths as necessary bearing the constraints you're assuming in mind
In general, changing data formats to something incompatible is one of the most expensive changes you can possibly make to your software. If you’re not sure write a prototype and profile. Implementing knowingly inefficient data format for a performance-critical application isn’t a good idea.
> it will only cause marginal harm to do everything in byte offsets and shift and OR bytes to yield final values
Huge harm, in both runtime performance, and code size & complexity.
P.S. For applications where portability matters and you’re OK paying performance cost of that, the industry has moved towards XML based formats. Not only it fixes byte order issues, also text encodings, globalization, it’s human readable, and it’s fast enough for many practical applications. E.g. doc/xls that you’ve mentioned are deprecated by docx/xlsx, the latter are XML based.
If:
* you're working with files that only your software will access AND
* you control, and understand, the CPU architecture of any machine that will touch that data structure and you KNOW that it will never change for the entire lifetime of that piece of data AND
* you always use the same version, or an ABI-compatible future version, of the same compiler that you KNOW will always lay out data the same way and you know what that way is AND
* you either don't need to touch this data structure in other programming languages, or you KNOW that this won't be an issue (for example because your programming language implementation was written in C, compiled against the same version of the compiler, and has an FFI that understands C structs)
THEN, you may proceed to mmap structs into memory. In practice these constraints are fairly commonplace; for example, NetBSD wscons device drivers report HID events which are specified in a struct, and because the HID events are likely never to leave the originating machine, only be passed from kernel space to user space, it makes sense to simply read(2) them straight into the struct.
And there's certainly nothing wrong with snarfing a large file into a char[], but when it comes time to extract meaning from it, unless you are absolutely sure that the underlying assumptions regarding data layout will never ever change, it will only cause marginal harm to do everything in byte offsets and shift and OR bytes to yield final values -- and this is the best portable way to access the data therein, agnostic of details about the compiler and CPU architecture.
In general it's a good idea to err on the side of caution by default, profile, and then optimize the hot paths as necessary bearing the constraints you're assuming in mind (and perhaps documenting them for good measure).
Maybe my initial statement was too strong, but I shudder whenever I see comments of the form "It's so convenient to just mmap() that sucker into a struct and access the fields!" Because they've never had to deal with the consequences of trying to access a struct from a Microsoft compiler serialized to disk, and finding out that GNU compilers have a quite different notion of how structure members are to be laid out in memory...