CodExt – Encode/decode anything with Python

CodExt is a (Python2-3 compatible) library that extends the native codecs library (namely for adding new custom encodings and character mappings), hence its name combining CODecs EXTension. It provides 120+ new codecs and also features a guessing mode for decoding multiple layers of encoding and CLI tools for convenience.

$ echo "CodExt, a Python library to encode/decode anything !" | codext encode manchester gzip base58 bcd lz77 base37
lzh2f4w9om6eytpu49lcbiyb5jccsigxm09r[...]

Setup

This library is available on PyPi and can be simply installed using Pip:

$ pip install codext

Why?

This package aims to be used as a library in other projects or to provide some useful tools for working from the terminal. It does not display fancy useless colors, it does just enough for IT pros to encode/decode anything. It leverages the neat API of the native codecs library, using a very flexible logic for defining and lookup up codecs.

Note that, while it is not designed to be full of nice-looking colorful tools, CodExt’s features overlap with some other projects:

• CyberChef, which is a great tool supporting many codecs but formatted as a Web application.
• Ciphey, handling less codecs but seemingly using a more performant Artificial Intelligence algorithm for performing multi-layer guessing.
• Katana, designed to cover a broader scope, especially for CTF’s, but proposing less codecs.
• Basecrack, providing a fancy colorful output but handling only a very limited set of base encodings and featuring a simplistic heuristic for multi-layer decoding.
• Sttr, which is a very handy tool written in Go with colored output and auto-completion but currently supports only a few codecs and has no guessing mode.

Features

CodExt can be used as a library or through its CLI tools.

API

Its API provides some new functions for working with codecs:

• add: for adding a custom encoding
• add_map: for adding a custom map encoding
• add_macro: for defining macros (chains of encodings)
• list: for listing available codecs
• remove: for removing codecs or macros

Multi-round encoding can be used thanks to the simple API:

>>> codext.encode("test string", "my-encoding[10]")
                                               ^
                                 apply 10 rounds of "my-encoding"

Macros can be very useful and handy. The following shows an example of what it looks like:

{
  "example-macro": [
    "gzip",
    "base62",
    "gzip",
    "base63",
    "gzip",
    "base64"
  ]
}

It is added to your macros.json local file with this simple function:

>>> codext.add("example-macro", "gzip", "base62", "gzip", "base63", "gzip", "base64"]

Or also this simple or command:

$ codext add-macro example-macro gzip base62 gzip base63 gzip base64

New codecs

CodExt features 120+ new codecs (in comparison, Ciphey proposes twice less) spread across multiple categories, for the sake of easy filtering (e.g. for use with the list function):

• bases 2, 3, 4, 8, 16, 32 (including Zbase32, hex, geohash), 36, 45, 58 (bitcoin, ripple, flickr), 62, 63, 64 (including URL), 67, 85, 91, 100, 122 and any generic base
• binary encodings like Baudot code, Binary Binary Coded Decimal (BCD), Excess3, Manchester, (bits-)Rotate
• common codecs like a1z26, octal, ordinal
• compression algorithms like GZIP, LZ77, LZ78, PKZIP
• cryptography-based dynamic codecs like Affine, Atbash, Bacon, Barbie, Citrix, Rot (Caesar), Scytale, Shift, XOR
• hashing-related codecs like MD4, MD5, SHA1, SH256, SHA512, SHA3_384, SHAKE, BLAKE
• languages like braille, leetspeak, morse code, Navajo, radio phonetic alphabet, southpark, tom-tom
• other particular encodings like DNA, letter indices, URL
• steganography codecs like Klopf (Polybius), resistor colors, SMS, whitespaces
• simple string manipulations like reversing characters or words, upper- or lowercase, etc

Guessing mode

CodExt provides a guessing mode relying on an artificial intelligence algorithm, a tree search (with pruning applied), using a score-based ranking heuristic for accelerating the research of the most relevant solution. It can be tuned with a stop-function, namely relying on languages (e.g. using langid or langdetect) or patterns (including a pre-defined “flag” pattern matching). This can be particularly useful during CTF’s.

Multiple parameters allow to tune the algorithm in addition of the stop function, including the minimum or maximum search depth, a list of codecs categories to be considered, and a starting list of already-known encodings. For some examples of usage, you can refer to this page.

CLI tools

CodExt also features multiple handy CLI tools:

• codext : the main tool, providing multiple commands for manipulating text

$ echo "A test string" | codext encode morse
.- / - . ... - / ... - .-. .. -. --.
$ echo "A test string" | codext encode morse pkzip_lzma base58-bitcoin
9jAzuEHUAWV3pvMqfMckct3JzSpf8oHhDSZcd2XboG67GtV74Y7kZ3BBd

$ echo "9jAzuEHUAWV3pvMqfMckct3JzSpf8oHhDSZcd2XboG67GtV74Y7kZ3BBd" | codext decode base58_bc pkzip-lzma morse
a test string

• baseX : multiple base tools that mimic and replace the Linux ones base32 and base64, including 2, 3, 4, 8, 16, 32, 32-z, 32-hex, 32-geohash, 36, 45, 58-bitcoin, 58-ripple, 58-flickr, 62, 63, 64, 64-url 67, 85, 91, 100, 122

$ echo "Another test string" | base85 | base58-ripple | base91 | base26
DIYNLCJWOELBHMDRTGAYUIEXKBNVXIALYYEDHTBCYLYIZVJOYMZTIFNQVVTLCISQRYDIUI

$ echo "DIYNL...DIUI" | base26 -d | base91 -d | base58-ripple -di | base85 -d
Another test string

• unbase : a tool for decoding multiple layers of base encodings using an AI algorithm

$ echo "Test string" | base64 | unbase
Test string
$ echo "Test string" | base32-geohash | unbase
Test string
$ echo "Test string" | base32-geohash | base64 | unbase
Test string
$ echo "Test string" | base63 | base26 | base91 | unbase
Test string

Contributing

Check this guide if you want to know how to add a new codec or you can also add a macro by proposing it through the related JSON file.