In Compressing AIS strings, part 2, we looked at how AIS strings are structured and whether they could be stored more efficiently.
The result of all that is a tool to compress text-based AIS strings into binary streams and back, with about a 29% reduction in data size in the process: github.com/JaimeValdemoros/ais-compact.
This can be installed directly using the cargo build tool:
cargo install --git https://github.com/JaimeValdemoros/ais-compact.git#0.1.0
It provides ais-compress and ais-decompress binaries, that will transfer AIS strings completely losslessly (even if the strings are malformed or contain garbage):
echo '!AIVDM,1,1,,A,13HOI:0P0000VOHLCnHQKwvL05Ip,0*23' | ais-compress | ais-decompress
# !AIVDM,1,1,,A,13HOI:0P0000VOHLCnHQKwvL05Ip,0*23
And since it works with STDIN and STDOUT, you can combine it with other tools to send data over the network. For example, using nc:
# Receiver (bind to all interfaces, port 10110)
# Get received data, decompress it, and write it to ./nmea-data
nc -l 0.0.0.0 10110 | ais-decompress > ./nmea-data
# Sender (connect to receiver over network in this case 127.0.0.1:10110)
# Get AIS data from (hypothetical) generate-ais-data command
# compress it, and send it over the connection
# '-N' instructs nc to close the connection when the input is done
generate-ais-data | ais-compress | nc -N 127.0.0.1 10110
Why did I build this?
I have a Raspberry Pi with an AIS receiver set up near the coast, picking up AIS signals from ships out at sea (see Experiments with dAISy).
I’m sharing that data with a couple of AIS collaboration services (see Sharing AIS for fun and profit), but as a result I’m using N times as much data sending N copies of the same data stream.
The Raspberry Pi uses a 4G SIM to transfer data, so it’s in my interest to keep the data usage as low as possible.
A more efficient use of the data allowance would be to stream a single copy to a computer back home, and from there farm out the data to wherever I want it to go.
Once i’ve got a server in play, I’ve now got a device either side of the 4G connection, which means I can experiment with different compressions schemes, without relying on 3rd parties being able to decompress the data.
Compressing AIS strings, part 1 and Compressing AIS strings, part 2 were the result of this investigation, where I found that simply compacting the AIS strings into more efficient binary formats provided a nifty 29% reduction in space usage.
Thus, ais-compact was born.
What’s next?
29% is a good starting point, but the experiments with zstd found that there’s a lot more space-saving available in larger blocks.
Any solution needs to allow live-streaming of data (i.e. messages shouldn’t be held up waiting for a batch to be ready to send), so that it’s more immediately useful (and to stay within some services’ requirement of max 60 second delay).
One technique would be to keep a cache of the last few minutes worth of messages on both the client and server, spot duplicates as they come in, and send over a reference instead of a whole message.
I’m sure there’s more patterns to be found - it’s a good excuse to explore the state of the art in compression algorithms!
Given sufficiently efficient compression, this might make it feasible to run an AIS receiver off a very low-cost IOT SIM, the sort that offers you 100MB a month.