uiua-bin

uiua-bin

Extensive binary (de-) serialization framework

Overview

Normal parsers have a signature of: RemainingBytes Value ? Bytes

There are also "stateful" parsers, that have a signatue of: RemainingBytes Value State ? Bytes State

All parsers should have a un-inverse.

All parsers that contain inner parsers should accept stateful parsers.

Getting started:

B ~ "git: github.com/alex-s168/uiua-bin"

~QoiHeader[Width Height Channels Colorspace]

QoiHeaderBin ← B~MS‼(
QoiHeader
| B~Just!(-@0"qoif")
| Width = B~Ube₃₂
| Height = B~Ube₃₂
| Channels = B~Ube₈
| Colorspace = B~Ube₈
)

QoiHeader 123 321 1 2
°QoiHeaderBin[]
# [113 111 105 102 0 0 0 123 0 0 1 65 1 2]

# pop is used to remove remaining bytes of parser
◌QoiHeaderBin

NoInv!

When writing serializers for variants, and som variant cases only support de-serialization, you might want to use NoInv! F to get a runtime error when the inverse is called, instead of a compile time error.

modifying binary data

Uiua provides the "under" modifier, which can be used for this:

[113 111 105 102 123 0 0 0 65 1 0 0 1 2]
# reduce "Width" in "QoiHeader" by 10
⍜(QoiHeader~Width◌QoiHeaderBin)(-10)
## [113 111 105 102 113 0 0 0 65 1 0 0 1 2]

Performance

The performance of most things is quite good, because array slicing (take and drop) is zero-cost.

Bit parsers however convert all bytes to bits first, and then convert the remaining bits back to bytes. This is a issue when you have lots of data. Because of this, consider using Sub!!

Bit

Bit module

Parsers that only work inside a bit context.

See Bits!


Bit~Ube! index macro

unsigned integer

argument is the number of bits

Example:

|Header {Width Height Channels Colorspace}

HeaderBin ← MS!(
Header
| Just!(-@\0"megic")
| Width = Ube₃₂
| Height = Ube₃₂
| Sub!!1 Bits! Compose!(
| Channels = Bit~Ube!4
| Colorspace = Bit~Ube!4
)
)
Source code
Ube!(
(^0|°^0)
| ˜( 0^0 )
)

Code macros

Compose! code macro

Consider using M! instead.

This is useful for when you are parsing a struct, have nesting, and want the nice syntax to replace F!!

Example:

|QoiHeader {Width Height Channels Colorspace}

QoiHeaderBin ← M!(
Z!QoiHeader
| Just!(-@\0"qoif")
| QoiHeader~Width = Ube₃₂
| QoiHeader~Height = Ube₃₂
| Sub!!2 Compose!(
| QoiHeader~Channels = Ube₈
| QoiHeader~Colorspace = Ube₈
)
)
Source code
Compose! ←^ (ComposeGen (GenF""))

M! code macro

Helper to create a compound data parser.

Please read to the end.

Example:

HeaderBin = M!(Z!Header
| F!!Ube,32 Header~Abc
)

This can also be used to parse tuples:

Test = M!([0 0]
| F!!Ube,32 pick,0
| F!!Ube,32 pick,1
)

There is also comptime syntax so you don't have to type F!!:

|QoiHeader {Width Height Channels Colorspace}

QoiHeaderBin ← M!(
Z!QoiHeader
| Just!(-@\0"qoif")
| QoiHeader~Width = Ube₃₂
| QoiHeader~Height = Ube₃₂
| QoiHeader~Channels = Ube₈
| QoiHeader~Colorspace = Ube₈
)

Also consider using MS!, for even cleaner code.

Source code
M! ←^ $"MCtx!!((_)|(_))" (
| ComposeGen(GenF"") (1)
)

MS! code macro

Read the documentation for M! first.

Example:

|QoiHeader {Width Height Channels Colorspace}

QoiHeaderBin ← B~MS!(
QoiHeader
| Just!(-@\0"qoif")
| Width ← Ube₃₂
| Height ← Ube₃₂
| Channels ← Ube₈
| Colorspace ← Ube₈
)
Source code
MS! ←^ $"MCtx!!(C~Z!(_)|(_))" (
| ComposeGen(GenF) ¤$"_!" (|(1))
)

V! code macro

decode a prefixed variant

example:

V!(
Ule₈
| 0 => Ule₁₆
| . => Ube₁₆
)

if the variant ID is not a prefix, consider using this in combination with Ref!

the value of this parser depends on which variant is chosen

It uses cursed.ua for parsing the match arms

Source code
V! ←^ EmitV

Index macros

Bits! index macro

Runs the given parser in a bit context.

This means that it will input and output bits instead of bytes

Converts all input bytes to bits, applies the parser, and then converts the remaining bits back to bytes.

for performance reasons, consider limiting the resulting parser using Sub!

inside a struct context, consider using this with Compose!

Source code
Bits!(
() # convert to bits
^0 # apply parser
PadL!8 # pad left to multiple of 8 bits
(°)¤¤8 # convert back to bytes
| ˜(
°^0[]
PadL!8 # pad left to multiple of 8 bits
(°)¤¤8 # convert to bytes
)
)

PaddingB! index macro

insert ^0 bytes that can be any value / padding bytes

Source code
PaddingB!(^0|^0 0)

Len! index macro

applies the parser, but pushes the amount of bytes it consumed / created onto the stack.

this can not be un-ed, but you can use UnLen!

Source code
Len!-:^0.

UnLen! index macro

Source code
UnLen!¯Len!°^0

PadR‼ index macro

pad output of parser ^1 to ^0 bytes with trailing zeros

Source code
PadR‼(
-:^0 Len!^1
| ::0-:^0 :(UnLen!^1[]))

F‼ index macro

assign the value of parser ^0 to struct field ^1

you can use this in M!!

Source code
F‼(
(^1():)^0
| °^0^1)

Just! index macro

a parser that encodes / matches and returns ^0

example:

Just!(-@\0"png")
Source code
Just!(
^0"magic mismatch"^0 ^0
| ^0 )

Arr‼ index macro

parse length-prefixed array.

macro arguments:

  1. parser for array len prefix
  2. parser for array elem

if the array length is not a prefix, consider using this in combination with Ref!

you might want to box the value of the inner parser using Map!!

Source code
Arr‼(
[]:^0
# Len Arr RemBytes ...

(
# Arr RemBytes ...
: ^1
# Arr Elt RemBytes ...
)
:
| :
# Arr Bytes ...
(°^1:)
# Arr Bytes ...
°^0:
)

NArr‼ index macro

parses an array of a fixed length

macro arguments:

  1. array length (constant)
  2. parser for array elem

example:

F!! NArr!!4 Ule,8
Source code
NArr‼(
(˜^1)^0 []
| (°^1 (|¯ 1))^0
)

UntilEnd! index macro

applies the given parser (to generate an array) until no more bytes are left

Source code
UntilEnd!(
[]
# Arr Bytes ...
(^0
# Arr RemBytes Val ...
:
| >0
)
:
| :
# Arr Bytes ...
(°^0:)
)

Z! index macro

This function is deprecated and will be removed in the future.

Either use Z! from cursed.ua directly, or use MS!!

Source code
Z!C~Z!^0

MCtx‼ index macro

StructParser?ZeroInitFieldSerial

Helper to create a compound data parser.

YOU PROBABLY DON'T WANT TO USE THIS DIRECTLY. USE M! INSTEAD

Example:

HeaderBin = MCtx!!Z!Header(
F!!Ube,32 Header~Abc
)

This can also be used to parse tuples:

Test = MCtx!![0 0](
F!!Ube,32 pick,0
F!!Ube,32 pick,1
)
Source code
MCtx‼(^1^0|°^1)

Map‼ index macro

Parser?MapFnParser

modify value after it is parsed. also works with un by modifying the value before it is encoded.

Example:

Map!!box Arr!!Ule,16 U,8
Source code
Map‼(^0^1|°^1°^0)

FConv‼ index macro

Source code
FConv‼C~NoInv!(^0|Float~FloatLe!^1^0)

Var‼ index macro

decode a prefixed variant type

parsers of variants can have any amount of outputs

example:

#if u8 is 0, decode LE u16, otherwise BE u16
Var‼Ule₈C~InvTry!(
VarEnt‼0 Ule₁₆
| VarEnt‼1 Ube₁₆
| VarFail)

generated parser signature: RemBytes Variant Value ? Bytes

you might want to put this into Multi!

YOU PROBABLY WANT TO USE``V!!` INSTEAD

Source code
Var‼(: ^1 : ^0|°^0:°^1)

VarEnt‼ index macro

Source code
VarEnt‼(^0 ^1 °^0|^0°^1°^0:)

Ref! index macro

A parser that returns the value from the struct field, (or sets the value from the struct field in encode)

Works only inside a M!! context

this can be used together with other parsers, such as Arr!!, when the length is not a prefix:

~Struct { CoolArr CoolArrLen }

M!!Z!Struct(
F!! Ule,32 Struct~CoolArrLen
...
F!! Arr!! Ref! Struct~CoolArrLen U,8 Struct~CoolArr
)
Source code
Ref!(^0|°^0)

SimpleN‼ index macro

Parser?NumSimpleParser

takes in a parser of signature Value ? Bytes, and parses the first N bytes with it

Source code
SimpleN‼(
(^0|^1 ^0)
| :(°^0. ^1))

SimpleAll! index macro

takes in a parser of signature Value ? Bytes, and parsers all remaining bytes with it

Source code
SimpleAll!([]^0|:°^0)

Sub‼ index macro

Parser?NumParser

Runs the given parser in a sub-view of the bytes of length N.

This is useful for performance of bit parsers and also for until-end parsers.

Source code
Sub‼(
(^0|^1^0)
# RemBytesTail RemBytesHead Value ...
˜
| (°^1[])
# OldBytes AddBytes ...
˜
)

Noadic functions

VarFail function

|0.0
Source code
VarFail(VarFailImpl|VarFailImpl)

Monadic functions

Ule₈ function

|1.2Output1Output2?Input

This function is deprecated and will be removed in the future.

Use U,8 instead!

Source code
Ule₈UleConv!1

Ule₁₆ function

|1.2Output1Output2?Input
Source code
Ule₁₆UleConv!2

Ule₂₄ function

|1.2Output1Output2?Input
Source code
Ule₂₄UleConv!3

Ule₃₂ function

|1.2Output1Output2?Input
Source code
Ule₃₂UleConv!4

Ule₄₀ function

|1.2Output1Output2?Input
Source code
Ule₄₀UleConv!5

Ule₄₈ function

|1.2Output1Output2?Input
Source code
Ule₄₈UleConv!6

Ule₅₆ function

|1.2Output1Output2?Input
Source code
Ule₅₆UleConv!7

Ule₆₄ function

|1.2Output1Output2?Input
Source code
Ule₆₄UleConv!8

Ile₈ function

|1.2Output1Output2?Input

This function is deprecated and will be removed in the future.

Use I,8 instead!

Source code
Ile₈IleConv!1

Ile₁₆ function

|1.2Output1Output2?Input
Source code
Ile₁₆IleConv!2

Ile₂₄ function

|1.2Output1Output2?Input
Source code
Ile₂₄IleConv!3

Ile₃₂ function

|1.2Output1Output2?Input
Source code
Ile₃₂IleConv!4

Ile₄₀ function

|1.2Output1Output2?Input
Source code
Ile₄₀IleConv!5

Ile₄₈ function

|1.2Output1Output2?Input
Source code
Ile₄₈IleConv!6

Ile₅₆ function

|1.2Output1Output2?Input
Source code
Ile₅₆IleConv!7

Ile₆₄ function

|1.2Output1Output2?Input
Source code
Ile₆₄IleConv!8

Ube₈ function

|1.2Output1Output2?Input

This function is deprecated and will be removed in the future.

Use U,8 instead!

Source code
Ube₈UleConv!1

Ube₁₆ function

|1.2Output1Output2?Input
Source code
Ube₁₆BeConv‼2Ule₁₆

Ube₂₄ function

|1.2Output1Output2?Input
Source code
Ube₂₄BeConv‼3Ule₂₄

Ube₃₂ function

|1.2Output1Output2?Input
Source code
Ube₃₂BeConv‼4Ule₃₂

Ube₄₀ function

|1.2Output1Output2?Input
Source code
Ube₄₀BeConv‼5Ule₄₀

Ube₄₈ function

|1.2Output1Output2?Input
Source code
Ube₄₈BeConv‼6Ule₄₈

Ube₅₆ function

|1.2Output1Output2?Input
Source code
Ube₅₆BeConv‼7Ule₅₆

Ube₆₄ function

|1.2Output1Output2?Input
Source code
Ube₆₄BeConv‼8Ule₆₄

Ibe₈ function

|1.2Output1Output2?Input

This function is deprecated and will be removed in the future.

Use I,8 instead!

Source code
Ibe₈IleConv!1

Ibe₁₆ function

|1.2Output1Output2?Input
Source code
Ibe₁₆BeConv‼2Ile₁₆

Ibe₂₄ function

|1.2Output1Output2?Input
Source code
Ibe₂₄BeConv‼3Ile₂₄

Ibe₃₂ function

|1.2Output1Output2?Input
Source code
Ibe₃₂BeConv‼4Ile₃₂

Ibe₄₀ function

|1.2Output1Output2?Input
Source code
Ibe₄₀BeConv‼5Ile₄₀

Ibe₄₈ function

|1.2Output1Output2?Input
Source code
Ibe₄₈BeConv‼6Ile₄₈

Ibe₅₆ function

|1.2Output1Output2?Input
Source code
Ibe₅₆BeConv‼7Ile₅₆

Ibe₆₄ function

|1.2Output1Output2?Input
Source code
Ibe₆₄BeConv‼8Ile₆₄

U₈ function

|1.2Output1Output2?Input
Source code
U₈°˜

I₈ function

|1.2Output1Output2?Input
Source code
I₈IleConv!1

NullTerm function

|1.2Output1Output2?Input

null terminated byte array

Source code
NullTerm(
(+1) =0.
| :0:)

CStr₈ function

|1.2Output1Output2?Input

null terminated utf8 string

Source code
CStr₈°utf NullTerm

FLe₁₆ function

|1.2Output1Output2?Input
Source code
FLe₁₆FConv‼2 Float~FloatType~F₁₆

FLe₃₂ function

|1.2Output1Output2?Input
Source code
FLe₃₂FConv‼4 Float~FloatType~F₃₂

FLe₆₄ function

|1.2Output1Output2?Input
Source code
FLe₆₄FConv‼8 Float~FloatType~F₆₄

AMDFLe₂₄ function

|1.2Output1Output2?Input

AMD's 24 bit float format

Source code
AMDFLe₂₄FConv‼3 Float~FloatType~AMDF₂₄

BFLe₁₆ function

|1.2Output1Output2?Input

BrainFloat 16

Source code
BFLe₁₆FConv‼2 Float~FloatType~BF₁₆

FBe₁₆ function

|1.2Output1Output2?Input
Source code
FBe₁₆BeConv‼2 FLe₁₆

FBe₃₂ function

|1.2Output1Output2?Input
Source code
FBe₃₂BeConv‼4 FLe₃₂

FBe₆₄ function

|1.2Output1Output2?Input
Source code
FBe₆₄BeConv‼8 FLe₆₄

AMDFBe₂₄ function

|1.2Output1Output2?Input

AMD's 24 bit float format

Source code
AMDFBe₂₄BeConv‼3 AMDFLe₂₄

BFBe₁₆ function

|1.2Output1Output2?Input

BrainFloat 16

Source code
BFBe₁₆BeConv‼2 BFLe₁₆

None function

|1.2Output1Output2?Input

parses nothing

Source code
NoneJust![]