uiua-bin
Extensive binary (de-) serialization framework
Overview
Normal parsers have a signature of:
RemainingBytes Value ? Bytes
There are also "stateful" parsers, that have a signatue of:
RemainingBytes Value State ? Bytes State
All parsers should have a un-inverse.
All parsers that contain inner parsers should accept stateful parsers.
Getting started:
B ~ "git: github.com/alex-s168/uiua-bin"
~QoiHeader[Width Height Channels Colorspace]
QoiHeaderBin ← B~MS‼(
QoiHeader
| B~Just!(-@0"qoif")
| Width = B~Ube₃₂
| Height = B~Ube₃₂
| Channels = B~Ube₈
| Colorspace = B~Ube₈
)
QoiHeader 123 321 1 2
°QoiHeaderBin[]
# [113 111 105 102 0 0 0 123 0 0 1 65 1 2]
# pop is used to remove remaining bytes of parser
◌QoiHeaderBin
NoInv!
When writing serializers for variants, and som variant cases only support de-serialization,
you might want to use NoInv! F
to get a runtime error when the inverse is called, instead of a compile time error.
modifying binary data
Uiua provides the "under" modifier, which can be used for this:
[113 111 105 102 123 0 0 0 65 1 0 0 1 2]
# reduce "Width" in "QoiHeader" by 10
⍜(QoiHeader~Width◌QoiHeaderBin)(-10)
## [113 111 105 102 113 0 0 0 65 1 0 0 1 2]
Performance
The performance of most things is quite good, because array slicing (take and drop) is zero-cost.
Bit parsers however convert all bytes to bits first, and then convert the remaining bits back to bytes.
This is a issue when you have lots of data.
Because of this, consider using Sub!!
Bit
Bit module
Parsers that only work inside a bit context.
See Bits!
Bit~Ube! index macro
unsigned integer
argument is the number of bits
Example:
|Header {Width Height Channels Colorspace}
HeaderBin ← MS!(
Header
| Just!(-@\0"megic")
| Width = Ube₃₂
| Height = Ube₃₂
| Sub!!1 Bits! Compose!(
| Channels = Bit~Ube!4
| Colorspace = Bit~Ube!4
)
)
Source code
Ube! ← ⌅( ⊃(↘^0|°⋯⇌↙^0) | ˜⊂⊙(⇌ ⬚0↙^0 ⋯) )
Code macros
Compose! code macro
Consider using M!
instead.
This is useful for when you are parsing a struct,
have nesting, and want the nice syntax to replace F!!
Example:
|QoiHeader {Width Height Channels Colorspace}
QoiHeaderBin ← M!(
Z!QoiHeader
| Just!(-@\0"qoif")
| QoiHeader~Width = Ube₃₂
| QoiHeader~Height = Ube₃₂
| Sub!!2 Compose!(
| QoiHeader~Channels = Ube₈
| QoiHeader~Colorspace = Ube₈
)
)
Source code
Compose! ←^ (ComposeGen ≡(GenF""))
M! code macro
Helper to create a compound data parser.
Please read to the end.
Example:
HeaderBin = M!(Z!Header
| F!!Ube,32 Header~Abc
)
This can also be used to parse tuples:
Test = M!([0 0]
| F!!Ube,32 pick,0
| F!!Ube,32 pick,1
)
There is also comptime syntax so you don't have to type F!!
:
|QoiHeader {Width Height Channels Colorspace}
QoiHeaderBin ← M!(
Z!QoiHeader
| Just!(-@\0"qoif")
| QoiHeader~Width = Ube₃₂
| QoiHeader~Height = Ube₃₂
| QoiHeader~Channels = Ube₈
| QoiHeader~Colorspace = Ube₈
)
Also consider using MS!
, for even cleaner code.
Source code
M! ←^ $"MCtx!!((_)|(_))" ⊃( ⊢⊙◌| ComposeGen≡(GenF"") ∩(↘1))
MS! code macro
Read the documentation for M!
first.
Example:
|QoiHeader {Width Height Channels Colorspace}
QoiHeaderBin ← B~MS!(
QoiHeader
| Just!(-@\0"qoif")
| Width ← Ube₃₂
| Height ← Ube₃₂
| Channels ← Ube₈
| Colorspace ← Ube₈
)
Source code
MS! ←^ $"MCtx!!(C~Z!(_)|(_))" ⊃( ⊢⊙◌| ComposeGen≡(GenF) ¤$"_!" ⊃(⊢⊙◌|∩(↘1)))
V! code macro
decode a prefixed variant
example:
V!(
Ule₈
| 0 => Ule₁₆
| . => Ube₁₆
)
if the variant ID is not a prefix,
consider using this in combination with Ref!
the value of this parser depends on which variant is chosen
It uses cursed.ua for parsing the match arms
Source code
V! ←^ EmitV
Index macros
Bits! index macro
Runs the given parser in a bit context.
This means that it will input and output bits instead of bytes
Converts all input bytes to bits, applies the parser, and then converts the remaining bits back to bytes.
for performance reasons, consider limiting the resulting parser using Sub!
inside a struct context, consider using this with Compose!
Source code
Bits! ← ⌅( ♭≡(⇌⋯₈) # convert to bits ^0 # apply parser PadL!8 # pad left to multiple of 8 bits ⧈(°⋯⇌)¤¤8 # convert back to bytes| ˜⊂⊙( °^0[] PadL!8 # pad left to multiple of 8 bits ⧈(°⋯⇌)¤¤8 # convert to bytes ))
PaddingB! index macro
insert ^0
bytes that can be any value / padding bytes
Source code
PaddingB! ← ⌅(↘^0|⊂↯^0 0)
Len! index macro
applies the parser, but pushes the amount of bytes it consumed / created onto the stack.
this can not be un-ed, but you can use UnLen!
Source code
Len! ← -:⊙⊸⧻⊙^0⧻.
UnLen! index macro
Source code
UnLen! ← ¯Len!°^0
PadR‼ index macro
pad output of parser ^1
to ^0
bytes with trailing zeros
Source code
PadR‼ ← ⌅( ↘-:^0 Len!^1| ⊂:⊂↯:0-:^0 :⊙(UnLen!^1[]))
F‼ index macro
assign the value of parser ^0
to struct field ^1
you can use this in M!!
Source code
F‼ ← ⌅( ⊙(⍜^1(∘◌):)^0| °^0⊙⊸^1)
Just! index macro
a parser that encodes / matches and returns ^0
example:
Just!(-@\0"png")
Source code
Just! ← ⌅( ⊙^0⍤"magic mismatch"≍^0 ⊃↙↘⧻^0| ⊂^0 ⊙◌)
Arr‼ index macro
parse length-prefixed array.
macro arguments:
- parser for array len prefix
- parser for array elem
if the array length is not a prefix, consider using this in combination with Ref!
you might want to box the value of the inner parser using Map!!
Source code
Arr‼ ← ⌅( ⊙[]:^0 # Len Arr RemBytes ...
⍥( # Arr RemBytes ... ⊙: ⊙^1 # Arr Elt RemBytes ... ⊂ ) :| : # Arr Bytes ... ⟜∧(°^1:)⇌ # Arr Bytes ... °^0:⧻)
NArr‼ index macro
parses an array of a fixed length
macro arguments:
- array length (constant)
- parser for array elem
example:
F!! NArr!!4 Ule,8
Source code
NArr‼ ← ⌅( ⍥(⊙˜⊂^1)^0 ⊙[]| ⊙◌ ⍥(°^1 ⊙⊃(⊣|↘¯ 1))^0)
UntilEnd! index macro
applies the given parser (to generate an array) until no more bytes are left
Source code
UntilEnd! ← ⌅( [] # Arr Bytes ... ⍢(⊙^0 # Arr RemBytes Val ... ⊂⊙: | >0⋅⧻ ) :| : # Arr Bytes ... ∧(°^0:)⇌)
Z! index macro
This function is deprecated and will be removed in the future.
Either use Z!
from cursed.ua directly,
or use MS!!
Source code
Z! ← C~Z!^0
MCtx‼ index macro
Helper to create a compound data parser.
YOU PROBABLY DON'T WANT TO USE THIS DIRECTLY.
USE M!
INSTEAD
Example:
HeaderBin = MCtx!!Z!Header(
F!!Ube,32 Header~Abc
)
This can also be used to parse tuples:
Test = MCtx!
Source code
MCtx‼ ← ⌅(^1⊙^0|⊙◌°^1)
Map‼ index macro
modify value after it is parsed. also works with un by modifying the value before it is encoded.
Example:
Map!!box Arr!!Ule,16 U,8
Source code
Map‼ ← ⌅(⊙^0^1|°^1⊙°^0)
FConv‼ index macro
Source code
FConv‼ ← C~NoInv!⊃(↘^0|Float~FloatLe!^1↙^0)
Var‼ index macro
decode a prefixed variant type
parsers of variants can have any amount of outputs
example:
#if u8 is 0, decode LE u16, otherwise BE u16
Var‼Ule₈C~InvTry!(
VarEnt‼0 Ule₁₆
| VarEnt‼1 Ube₁₆
| VarFail)
generated parser signature:
RemBytes Variant Value ? Bytes
you might want to put this into Multi!
YOU PROBABLY WANT TO USE``V!!` INSTEAD
Source code
Var‼ ← ⌅(: ^1 : ^0|°^0:°^1)
VarEnt‼ index macro
Source code
VarEnt‼ ← ⌅(^0 ^1 °^0|^0°^1°^0:)
Ref! index macro
A parser that returns the value from the struct field, (or sets the value from the struct field in encode)
Works only inside a M!!
context
this can be used together with other parsers, such as Arr!!
, when the length is not a prefix:
~Struct { CoolArr CoolArrLen }
M!!Z!Struct(
F!! Ule,32 Struct~CoolArrLen
...
F!! Arr!! Ref! Struct~CoolArrLen U,8 Struct~CoolArr
)
Source code
Ref! ← ⌅(⊙⊸^0|⊙°⊸^0)
SimpleN‼ index macro
takes in a parser of signature Value ? Bytes
,
and parses the first N bytes with it
Source code
SimpleN‼ ← ⌅( ⊃(↘^0|^1 ↙^0)| ⊂:⊙(°^0⧻. ^1))
SimpleAll! index macro
takes in a parser of signature Value ? Bytes
,
and parsers all remaining bytes with it
Source code
SimpleAll! ← ⌅([]^0|⊂:⊙°^0)
Sub‼ index macro
Runs the given parser in a sub-view of the bytes of length N.
This is useful for performance of bit parsers and also for until-end parsers.
Source code
Sub‼ ← ⌅( ⊃(↘^0|^1↙^0) # RemBytesTail RemBytesHead Value ... ˜⊂| ⊙(°^1[]) # OldBytes AddBytes ... ˜⊂)
Noadic functions
VarFail function
Source code
VarFail ← ⌅(VarFailImpl|VarFailImpl)
Monadic functions
Ule₈ function
This function is deprecated and will be removed in the future.
Use U,8
instead!
Source code
Ule₈ ← UleConv!1
Ule₁₆ function
Source code
Ule₁₆ ← UleConv!2
Ule₂₄ function
Source code
Ule₂₄ ← UleConv!3
Ule₃₂ function
Source code
Ule₃₂ ← UleConv!4
Ule₄₀ function
Source code
Ule₄₀ ← UleConv!5
Ule₄₈ function
Source code
Ule₄₈ ← UleConv!6
Ule₅₆ function
Source code
Ule₅₆ ← UleConv!7
Ule₆₄ function
Source code
Ule₆₄ ← UleConv!8
Ile₈ function
This function is deprecated and will be removed in the future.
Use I,8
instead!
Source code
Ile₈ ← IleConv!1
Ile₁₆ function
Source code
Ile₁₆ ← IleConv!2
Ile₂₄ function
Source code
Ile₂₄ ← IleConv!3
Ile₃₂ function
Source code
Ile₃₂ ← IleConv!4
Ile₄₀ function
Source code
Ile₄₀ ← IleConv!5
Ile₄₈ function
Source code
Ile₄₈ ← IleConv!6
Ile₅₆ function
Source code
Ile₅₆ ← IleConv!7
Ile₆₄ function
Source code
Ile₆₄ ← IleConv!8
Ube₈ function
This function is deprecated and will be removed in the future.
Use U,8
instead!
Source code
Ube₈ ← UleConv!1
Ube₁₆ function
Source code
Ube₁₆ ← BeConv‼2Ule₁₆
Ube₂₄ function
Source code
Ube₂₄ ← BeConv‼3Ule₂₄
Ube₃₂ function
Source code
Ube₃₂ ← BeConv‼4Ule₃₂
Ube₄₀ function
Source code
Ube₄₀ ← BeConv‼5Ule₄₀
Ube₄₈ function
Source code
Ube₄₈ ← BeConv‼6Ule₄₈
Ube₅₆ function
Source code
Ube₅₆ ← BeConv‼7Ule₅₆
Ube₆₄ function
Source code
Ube₆₄ ← BeConv‼8Ule₆₄
Ibe₈ function
This function is deprecated and will be removed in the future.
Use I,8
instead!
Source code
Ibe₈ ← IleConv!1
Ibe₁₆ function
Source code
Ibe₁₆ ← BeConv‼2Ile₁₆
Ibe₂₄ function
Source code
Ibe₂₄ ← BeConv‼3Ile₂₄
Ibe₃₂ function
Source code
Ibe₃₂ ← BeConv‼4Ile₃₂
Ibe₄₀ function
Source code
Ibe₄₀ ← BeConv‼5Ile₄₀
Ibe₄₈ function
Source code
Ibe₄₈ ← BeConv‼6Ile₄₈
Ibe₅₆ function
Source code
Ibe₅₆ ← BeConv‼7Ile₅₆
Ibe₆₄ function
Source code
Ibe₆₄ ← BeConv‼8Ile₆₄
U₈ function
Source code
U₈ ← °˜⊂
I₈ function
Source code
I₈ ← IleConv!1
NullTerm function
null terminated byte array
Source code
NullTerm ← ⌅( ⊃(↘+1)↙ ⊢⊚=0.| ⊂⊂:0:)
CStr₈ function
null terminated utf8 string
Source code
CStr₈ ← ⊙°utf₈ NullTerm
FLe₁₆ function
Source code
FLe₁₆ ← FConv‼2 Float~FloatType~F₁₆
FLe₃₂ function
Source code
FLe₃₂ ← FConv‼4 Float~FloatType~F₃₂
FLe₆₄ function
Source code
FLe₆₄ ← FConv‼8 Float~FloatType~F₆₄
AMDFLe₂₄ function
AMD's 24 bit float format
Source code
AMDFLe₂₄ ← FConv‼3 Float~FloatType~AMDF₂₄
BFLe₁₆ function
BrainFloat 16
Source code
BFLe₁₆ ← FConv‼2 Float~FloatType~BF₁₆
FBe₁₆ function
Source code
FBe₁₆ ← BeConv‼2 FLe₁₆
FBe₃₂ function
Source code
FBe₃₂ ← BeConv‼4 FLe₃₂
FBe₆₄ function
Source code
FBe₆₄ ← BeConv‼8 FLe₆₄
AMDFBe₂₄ function
AMD's 24 bit float format
Source code
AMDFBe₂₄ ← BeConv‼3 AMDFLe₂₄
BFBe₁₆ function
BrainFloat 16
Source code
BFBe₁₆ ← BeConv‼2 BFLe₁₆
None function
parses nothing
Source code
None ← Just![]