I am sure that Tutanota does not use any custom encryption algorithm. It is clearly stated in the FAQ that they use RSA (with PFS) and AES to encrypt emails exchanged between Tutanota users. https://tutanota.com/encryption

These are only primitive algorithms, the actual implementation is custom and specific to Tutanota, which mean it will only work with Tutanota as nothing else will implement it.

There is no way to do key distribution outside of Tutanota’s service.