NOTE: We anticipate publishing a security release, Synapse 1.47.1, on the coming Tuesday, the 23rd of November.
Synapse 1.47.1 will contain a fix for a high severity issue.
Synapse 1.47.0 features additions to the admin and module APIs, a plethora of fixes for long-standing bugs, and a raft of internal improvements. Server administrators should note that this release removes a deprecated API for deleting a room and deprecates a module callback. Consult the upgrade notes for more details.
We fixed a variety of different bugs in this release, many of which were long-standing. It's good to see them dealt with! Worth mentioning in particular:
Improvements to thehandling of the device_inbox table, which trim redundant data to reduce database bloat.
Fixes related to restartingworkers, to ensure a more reliable upgrade process.
Additionally, work continues on improving type-checking coverage, both inSynapse and inSygnal.
Sydent 2.5.1
This week also saw the release of Sydent 2.5.1, the reference implementation of a Matrix Identity Server. This is a minor release which mainly tidies up error handling to reduce the amount of noise in logs. It should also make it easier for us to diagnose some outstanding bugs which remain to be squashed.
Everything Else
In the background, we're still working away at implementing MSC3440 to facilitate threading. Early tests are promising. We're also exploring MSC2775 as a means to speed up room joins. Both will be long term projects that should hopefully reap major rewards for the Matrix ecosystem. Lastly, there's support for MSC3228 to allow identity servers to provide bespoke invites to spaces. We mentioned this last time in Sydent release notes; now we've got support for it on the Synapse side.
Please see the Synapse Release Notes for a complete list of changes in this release.
Here's your weekly spec update! The heart of Matrix is the specification - and this is modified by Matrix Spec Change (MSC) proposals. Learn more about how the process works at https://spec.matrix.org/unstable/proposals.
Matrix v1.1 was released! Read the blog post here if you missed it; it summarises everything that's new in v1.1, as well as plans for the future. Now that the new spec build pipeline and release infrastructure is in place, we're aiming for roughly quarterly releases going forward. Thank you all for being so patient in the meantime!
Extensible events is something that has been a long time coming in Matrix. It unlocks so much potential, and is even currently being built on (see MSC3381 (polls)). Definitely one of the next big ticket items to tackle in the medium term.
Dan (aka callahad) is away this week, so let me report on his behalf.
We cut a release candidate for Synapse (1.47.0rc2, but see the changelog for rc1). It exposes new functionality for pluggable modules and new endpoints to the Admin API. We've fixed a bunch of long-standing bugs and continued to drive forward efforts to improve documentation and code quality. Thank you to all of our contributors!
With future releases in mind, we've been continuing work to support threading and E2EE application services. We've also been prototyping a new Admin API to remove users from all rooms belonging to a certain space.
Elsewhere, we've been doubling down on our effort to improve reliability and maintainability of our services as a whole. Sydent and Sygnal have a number of PRs in flight for both, aimed at improving type coverage and driving down error noise in the logs. We've drafted a blog post to summarise the process of type annotating mypy (keep your eyes out for that one). We also worked to make the matrix.org database more resilient, and made changes to improve the experience of rolling out upgrades to Synapse en masse.
Vermicularis is a script for forwarding messages from Päikky to Matrix. Päikky is a parent-teacher communication website and mobile app for daycare and pre-school. git clone https://k2c42.dy.fi/git/vermicularis.git and join #paikky:ellipsis.fi to give feedback
Päikky was created by a group of parents who wanted to make communication with daycare staff easier. They started a company and later sold it to Abilita. Päikky is currently used in 40+ municipalities in Finland and also in some private daycare companies too.
Päikky is also used for reserving daycare times for kids.
Sensitive flag for MSG and NICKSERV to hide it from network room (for AUTOCMD)
Improved STATUS for admin room and simple STATUS for network rooms
Small fixes
Not much going on this week. Hopefully the reconnect refactor doesn't cause any breakage as it fixed multiple issues that have been around since the inception of Heisenbridge.
I guess not. For that reason bug fixing in Nheko for the next release continues. We fixed a super annoying issue, where loading keys from the online backup could make Nheko stuck in a flickering mode with no way to interact with it anymore. You can now also click anywhere on a read receipt to open someones profile instead of just their avaitar, edited messages now also show as redacted, if only the original message got redacted and don't lose the reply in encrypted rooms, if the edit was sent by a client, that is not Nheko. You also used to get logged out after registration, which should finally be resolved. There were also a bazillion translation updates! Thank you, everyone who contributed to those.
We also finally merged the prettier video player, which also fixes a video playback issue on macOS. Try it out and give us feedback on it!
We are still hunting down some last bugs, but expect the release soon now. In the meantime, I did start summarizing the changes, if you want to see what will be in the next release: https://github.com/Nheko-Reborn/nheko/blob/master/CHANGELOG.md#090----unreleased
Message threading is coming to Element! If you haven’t yet, head to develop.element.io and ensure ‘Enable threading’ is turned on in Labs to test on the web.
We’re spinning up development on iOS & Android as we speak.
We’ll also be running the first Threads community testing session next Thursday (18th Nov) at 17:00 GMT. Come join us in #element-community-testing:matrix.org!
As well as the community testing on Threads, we’ll also be hosting our next testing session for Android on Wednesday (17th), with the time to be confirmed. If you’d like to be involved, join us in #element-community-testing:matrix.org.
Element Web
In the background, if you’ve been paying close attention to our issue tracker, you might have noticed lots of changes to our triage & issue workflows over the past couple of months. We’re continuing to iterate on these, extending the best learnings to iOS & Android too.
We recently started a project to improve the info architecture/layout of our apps, starting on the web first. We’re merging our first tweaks and experiments soon, so expect exciting things to be landing on develop!
We’re continuing to implement more Space creation & management support on iOS. We’ve added in Space creation screens (implemented in SwiftUI!) and we’re polishing inviting people to Spaces.
We’ve also been conducting user research to see what parts of the app are tripping users up, and working on fixes to various issues. Expect tweaks to land soon!
Otherwise, we’re also merging & testing release candidates for upcoming releases, merging several branches.
Element Android
We’ve been conducting the same user research on Android, watch this space for more tweaks & improvements coming soon!
Otherwise, we’re also fixing up some smaller issues specific to Android 12.
Full UIAA implementation means now you can register an account on any hs that allow
Configure default homeserver with options to add more than one.
Can use http when looking for homeserver
Enhanced invite list UX
Added logout in loading screen
Hide pinned space notification from home icon
Add option to select role on roomCreation
Added Invite/disinvite option in profile viewer
Bugs
Fix commands activating anywhere in the input
Fix duplicate and minus notification count
Fix links splitting across line mid-word
Find more about Cinny at https://cinny.in/
Join our channel at: #cinny:matrix.org
Github: https://github.com/ajbura/cinny
Twitter: https://twitter.com/@cinnyapp
A time-like command that will send end of an arbitrary command output and some other info (like exit status) to matrix room. Useful when you need to run something in terminal and get a ping when it's done.
Since the last ttm news in TWIM new options were added:
matrix auth with access token - useful for SSO when you don't have actual login/password pair
skip time info
skip html formatting (doubles allowed log size in message)
post full log output to matrix (with auto-shrinker to avoid "message is too big" error)
Version 2.2.0 adds support for authentication via access_tokens.
In addition to username/access_token, it is possible to authenticate using username/password and login(SSO) token.
Example usage is shown below:
"""
Example Usage:
random_user
!echo something
echo_bot
something
"""
import simplematrixbotlib as botlib
creds = botlib.Creds(
homeserver="https://example.org",
username="echo_bot",
access_token="syt_c2...DTJ",
)
bot = botlib.Bot(creds)
PREFIX = '!'
@bot.listener.on_message_event
async def echo(room, message):
match = botlib.MessageMatch(room, message, bot, PREFIX)
if match.is_not_from_this_bot() and match.prefix() and match.command(
"echo"):
await bot.api.send_text_message(room.room_id,
" ".join(arg for arg in match.args()))
bot.run()
Hello again! Halcyon is a Matrix bot library created with the intention of being easy to install and use.
This release brings minor non-breaking features and some bug fixes:
Added
change_presence() now allows you to set if you are online, idle, or away. Status message support
Roadmap and documentation updates
Fixed
A fix for retrying on 5xx errors
A better catch for bad server syncs
Windows support, fixing the NotImplementedError reported by @bhuitt (Thanks!)
More info at on the project at https://github.com/WesR/Halcyon . Come by and chat with us over in https://matrix.to/#/#halcyon:blackline.xyz
Heads up for those in Berlin. You're welcome to join us Tuesday, 16th Nov at 7:00 PM chatting about Matrix development and hosting. We're going to meet in person at c-base. In compliance with the hackerspace's house rules this is a strict 2G event.
We'll talk about everyone's Matrix projects, test the P2P demo via Bluetooth in person and plan the presence of Matrix at the rC3 event happening in the c-base shortly before New Years.
"Sign in with Matrix" is a web component, which developers can use to build a web login using Matrix account
it is similar to those "Sign in with Google" and "Sign in with Facebook" buttons you see on the internet
but now it's Matrix!
i invite you to take a look at the repository: https://github.com/mishushakov/signin-with-matrix
and experience a demo: https://mishushakov.github.io/signin-with-matrix/
Dept of Ping 🏓
Here we reveal, rank, and applaud the homeservers with the lowest ping, as measured by pingbot, a maubot that you can host on your own server.
Once again it's been a little while since we've done a spec release (sorry; we're aiming for quarterly releases from here on out), but this time we have some pretty big news: we've got an all-new spec platform and a new versioning scheme. The new spec platform has been needed for a long time to make better sense of what Matrix is, and as part of getting that out the door we reduced the number of "Matrix versions" to just one.
Huge thanks to Will Bamberg for building it out for us, anoa for working out the deployment details, and everyone for testing it all. They talk at length about what this specification even is and about the platform itself on Matrix Live S6E19. It's the single greatest improvement to the spec we've seen to date.
The new versioning scheme presents the whole specification as a single document, making it easier to check compatibility between implementations and the spec itself. Previously, a grid would have to be drawn to show whether Server-Server r0.1.4 is compatible with Client-Server r0.6.1 - while obvious at release time, the historical context gets lost quite easily. Now, with a single version number, the entire specification is compatible across a single version number: servers implementing Matrix 1.1 will be compatible with clients implementing v1.1, and vice versa. For the specification itself, this means we no longer have to carefully point and update links between the APIs, as they'll instead be versioned together.
Changing the versioning of the specification does have an impact on the Client-Server API in particular, however. You may have noticed that Client-Server APIs are currently versioned at "r0". By removing rX.Y.Z versioning, all of the endpoints suddenly don't have a version to reference. All endpoints across the specification are now versioned individually to allow for breaking changes at the endpoint level. They no longer require the whole specification to be listed as a breaking change: a v1 endpoint can get additions/changes which are backwards compatible, but if we need to change format (for example) then it'll get bumped up to v2, leaving v1 in its last known state.
For the Client-Server API, a slight complication is that v1 and v2 (alpha) are already versions familiar to those that have been around for a while - to avoid confusing those people, existing Client-Server API endpoints will start at v3. New endpoints introduced after v1.1 will start at v1 instead.
It's been well over a year since the last release, which means there's a whole lot of features and changes to cover. Here we'll try to cover the things most clients/servers will have to worry about, but we do still recommend reading through the changelog included below. Overall, 28 MSCs have been merged through this release, but some have had to be excluded in the interest of getting the spec release out there. Many of them are expected to be in the next anticipated release (which should hopefully be next quarter).
Clients: There's a lot of stuff
In Matrix 1.1, client developers get all sorts of new features to play with. Clients which support end-to-end encryption should explore key backups, cross-signing, SSSS, and breaking changes to verification. Quite a lot of this stuff has existed for a while, but has made it into the specification formally now. As an added bonus, the emoji for SAS verification have been translated (contribute here).
Knocking has also landed in the spec (thanks Sorunome for leading the charge on this!), opening up the ability for rooms to optionally allow people to request invites to join. This can be helpful for semi-private rooms where it can be easier to approve/deny requests compared to finding someone's MXID and manually inviting them. This does require a v7 room to work, however.
Thanks again to Sorunome, Message Spoilers have been officially included in Matrix. People can now safely discuss the ending to the latest movie without being banned for spoilers. Though, as a new feature, there's a chance that the spoiler text still gets included in the message: clients should update as soon as possible to avoid their users accidentally getting banned for spoiling the conclusion to the Spaces saga ;)
There's a few other smaller improvements/additions, and of course the regular "various clarifications and bug fixes" to take a look at. For a sample checklist, check out element-web's issue on the subject.
Servers: Knock knock, it's a new room version
Room version 7 has landed, bringing forth the ability for users to knock on rooms (requesting an invite to join). The changes are largely scoped to using the reserved keywords for join rules and membership, and are described through the auth rules. Thankfully, the changes over v6 are minimally invasive so should be quick to implement.
Additionally, the cross-signing bits have been included in the API responses and EDU definitions. Matthew's blog post from last year (it really has been that long...) describes cross-signing and the history of its implementation.
As per usual, there's also various specification errors corrected to aid understanding. Synapse has an exhaustive issue to detail what servers might need to do.
PS: Room versions 8 and 9 are also out there, but will be included in a future spec release.
The full changelog
We haven't mentioned identity servers, bridges, etc in this post but they have changes too! Below is the whole changelog, the entire year and a bit of it. Thank you to everyone who has submitted MSCs, and congratulations on getting them released. If we forgot yours, please mention it in #matrix-spec:matrix.org so we can apologize and correct.
PS: The MSC process is how changes to Matrix are made, and you (yes, you) can propose those changes too. Check out the Matrix Live episode where Matthew talks about how this process works, and how we avoid blocking clients from implementing their proposals behind the relatively slow spec release cycles.
Client-Server API
Breaking Changes
Document curve25519-hkdf-sha256 key agreement method for SAS verification, and deprecate old method as per MSC2630. (#2687)
Add m.key.verification.ready and m.key.verification.done to key verification framework as per MSC2366. (#3139)
Deprecations
Deprecate starting verifications that don't start with m.key.verification.request as per MSC3122. (#3199)
New Endpoints
Add key backup (/room_keys/*) endpoints as per MSC1219. (#2387, #2639)
Add POST /keys/device_signing/upload and POST /keys/signatures/upload as per MSC1756. (#2536)
Add User-Interactive Authentication fields to cross-signing APIs as per MSC1756. (#3331)
Mention that a canonical alias event should be added when a room is created with an alias. (#3337)
Add an 'API conventions' section to the Appendices. (#3350)
Clarify the documentation around the pagination tokens used by /sync, /rooms/{room_id}/messages, /initialSync, /rooms/{room_id}/initialSync, and /notifications. (#3353)
Remove the inaccurate 'Pagination' section. (#3366)
Clarify how redacted_because is meant to work. (#3411)
Remove extraneous mimetype from EncryptedFile examples, as per MSC2582. (#3412)
Describe how MSC2844 affects the /versions endpoint. (#3420)
Fix documentation errors around threepid_creds. (#3471)
Server-Server API
New Endpoints
Add /make_knock and /send_knock endpoints as per MSC2403. (#3154)
Backwards Compatible Changes
Add cross-signing information to GET /user/keys and GET /user/devices/{userId}, m.device_list_update EDU, and a new m.signing_key_update EDU as per MSC1756. (#2536)
This week we have demos from andybalaam, showcasing high contrast in Element Web/Desktop, Bruno walking us through device dehydration in Hydrogen and Kegan giving us a glimpse of Sync v3! A lot of exciting work going on here!
Here's your weekly spec update! The heart of Matrix is the specification - and this is modified by Matrix Spec Change (MSC) proposals. Learn more about how the process works at https://spec.matrix.org/unstable/proposals.
The spec release mentioned last week is happening on Tuesday - look out for a long blog post explaining all the changes, and check out the new platform (as it stands in its current, unstable state) before it goes live. Exciting stuff!
Other than this sounding like something out of a workout routine, soft kicks are proposed as a hint to clients that the user has been kicked from the room, but that it was not out of moderation, but for a technical reason (the example given disconnection from an IRC network). The client can then present a different UX as opposed to a normal, "hard" kick.
Give it a read and provide feedback if the topic interests you!
This week we released Synapse 1.46! This release includes a fix for the performance regression from 1.44 which could cause smaller hosts to lock up when clients reconnected after some time offline.
Synapse 1.46 also includes experimental support for MSC3440: Threading via m.thread relation, a big step toward threaded messaging in Matrix. It's otherwise mainly a bugfix release, as we've spent much of our time reviewing infrastructure and making plans to improve our resilience should we experience hardware failures similar to what took out matrix.org two weeks ago.
We continue working toward a prototype implementation of MSC2775: Lazy loading over federation, starting with setting up automated infrastructure to repeatedly measure the join speeds of assorted large, public rooms. Our goal is to have a rough proof of concept for lazy room joins by the end of the year.
We also released Sydent 2.5.0! In addition to fixing bugs, this release passes mypy --strict, uses Jinja2 for templates, and supports the room_type field from MSC3288 to better differentiate between invitations to rooms and invitations to Spaces.
Threads! Awesome! And lazy loading over the federation looks really promising, I can't wait for it.
Soru has not found any time to work on the mx-puppet-* project in very long. At this point, it makes sense to look for someone / multiple people to inherit the project to. So, if you are willing to pick it up, or part of it (like a single bridge, only the library, etc.), then please look by #mx-puppet-bridge:sorunome.de. Thanks!
It's always sad when maintainers don't have time for former projects, but you can take over!
mautrix-whatsapp v0.2.0 was released. The main change is switching to the new WhatsApp multidevice API, which means the phone doesn't need to be connected 24/7 anymore. The release also includes a new backfilling system using MSC2716, and making the relaybot mode less special-cased and more like what mautrix-signal does.
Allow opening multiple simultaneous connections to the same network
New STOP/STAHP ROOM command to end a long paste flood
Clear channel send queue if the user gets kicked to prevent continuing flood on rejoin
New KB ROOM command to do a proper kick & ban (contributed by GitHub user @blastrock, thanks!)
The most useful feature this week is having the ability to open multiple connections to the same network. This can be used to plumb channels with a secondary connection while using the bridge as yourself with the main one. You can also STAHP long pastes from being actually sent out completely to a channel if you weren't careful enough on Matrix side.
Upgrading from previous version to v1.5.0 will cause a small migration for every channel/pm that is open so it might take a little longer for the first startup to complete. If no multiple network connections are opened it is safe to downgrade but if someone uses the feature it will cause shenanigans for the duplicate networks in the older version.
We are preparing for a release! That means, if you are translator, we could use your help to get our translations updated and if you want to have a bug fixed before release, now is the time to report it or complain about it! Please come and see us in #nheko:nheko.im for complaints and questions!
We saw a lot of translation updates by Linerly, Priit and Thulinma already. The click ghosting issue in the room list should be fixed as well as the crash when accepting an invite from your browser. You can now register on matrix.org again, after fighting with some... reality deforming wording in the spec. Notifications on macOS now use the "modern" API for notifications, including grouping notifications by room. (Expect more fancy improvements there in the future!) The warning about mentioning the whole room now vanishes properly, hiding space children from the roomlist now doesn't hide them from the space itself anymore, the loading spinner shouldn't show in a fully loaded room again, the room directory search bar is now focused by default and a few smaller fixes.
We also completed the bootstrapping UI now. This means you get prompted for unverified devices, when your device isn't verified or when cross-signing isn't configured for your account yet and you are given ways to remedy that.
We have a few more bug fixes we want to land before the release, so don't expect it tomorrow, but we have the featureset down now. And if you thought the changes listed in this snippet are long, wait for the full release changelog. It will probably take a few days for me to tidy that up and I think this will be our biggest release yet! Happy testing, everyone!
Device dehydration support got released this week in 0.2.20 (see matrix live demo!), allowing you to decrypt messages sent to you while not logged in on any device. Expect another release this evening (0.2.21) with support for multiple lines in the composer and some other bug fixes.
After being away for a few month, Kévin Commaille came back with a blast. No less than three MR already, and I have a feeling he’s only getting started:
We’re continuing to flesh out space creation screens on iOS, improving the invite flows post creation for private and public spaces.
We’re continuing to iterate on the overall app layout on Web, to fix fundamental usability issues.
Expect to see some changes on develop soon, rearranging the top left corner of the app (user avatar, space names, context menu).
Expect to see larger experiments landing on develop soon too. We’ll share more info on these when they’re live, and everything will be opt out to not break existing workflows while we’re testing and experimenting.
Community testing session ran successfully on RC with a focus on voice messages and VoIP calls. We found significantly fewer issues and regressions than in the previous sessions. It was great to see 9 newcomers - thank you to all of our testers who helped out!
Improved our sentry analytics data to provide more useful detail.
Redesigned the device management section in the user settings. The new view makes it clearer how to rename devices and lets you verify them.
This week in labs (features in labs can be accessed at develop.element.io and nightly desktop builds):
Threads now has a more polished UI and users can now link to events in a thread. Notifications for threads messages are being worked on and should land very soon.
Release 1.3.6 is finally live on the PlayStore! Release 1.3.7 is expected for the beginning of next week.
We have started to implement threading of messages.
Draft for voice message has been delayed, we want to ensure that it’s working well before releasing it.
We are doing lots of internal migrations: Dagger to Hild, MvRx to Mavericks2, Rx to coroutines Flow.
And as always, we are trying to keep the app as stable as possible, and as reliable as possible
It’s also worth noting that the SDK 1.3.6 has some improvements in the login API, not needed for Element Android, but requested by other users of the SDK.
Godot-Matrix is a C++ module for connecting to Matrix from the Godot Game Engine.
I refurbished it and the changes made it upstream now ^_^, thx vurpo! :
https://www.matrix.org/docs/projects/sdk/godot-matrix/
Changes:
Fixed compilation for 3.x Godot branch
Support plain http too for local tests with Conduit
Respect port numbers (as per Matrix spec)
Substantial rewrite of the example so it works
Refurbishment made with love for the participants of FediJam 4, a weeklong gamedev competition with the focus on fediverse and federated multiplayer (via Matrix or any other means ^_^).
It starts this Sunday at 18:00 UTC. Chat
Unlike other "overlay" style crates it does not follow the sdk versions using its own version number or using features. So please make sure you use the right version of the SDK with it.
Also it now has support for rusttls, the ! at the start of a command now gets correctly checked and is updated to the latest sdk.
You can check it out at https://crates.io/crates/mrsbfh and https://github.com/MTRNord/mrsbfh
New release of the Commander of etke.cc backoffice is coming here!
New version brings new neat features and extends arsenal of the antispam tricks:
buymeacoffee listings integration - pretty basic, but that's just a first step! So now you can list subscribers, supporters and extras purchases directly in matrix room (spolier: next releases will bring notifications about new subscribers, supporters and purchases)
rate limiter for http forms - no one can bother you if you don't want that! So now you can configure specific rate limits per form
enhanced email validation (mostly used in the forms module) - you had spam hosts and emails list before, but now you have MX records checks, regular expressions check and some other useful tricks to hold the door stop spammers
... and some small changes, like doc updates and auto-shrink of long messages to fit into matrix payload limits
I created a Vacation Responder for Matrix. While running, it responds to any DMs that come in to your account with a preset message. It can be configured to only respond once in a given period of time (defaults to 24 hours) so that the person contacting you can queue up questions for you when you return. Other ways to use the bot include responding to DMs to previously active accounts (for example if you move homeservers). Join the discussion about this bot at #matrix-vacation-responder:nevarro.space.
SeaGL (the Seattle GNU/Linux Conference) is being held on Matrix Friday and Saturday, November 5th and 6th, from 9am-5pm Pacific Time. The conference is running on a customized version of element-web and a regional Matrix homeserver, found at https://attend.seagl.org
Founded in 2013, this is the conference's ninth year overall and second virtual event. Last year, the all-volunteer team built their portal on-top of KiwiIRC and Jitsi for attendees, and BigBlueButton for speakers. This year they have been relying heavily on Spaces, Subspaces, and the built-in features that have been rapidly developing within the Matrix protocol.
From their website:
SeaGL (the Seattle GNU/Linux Conference) is a free—as in freedom and tea—grassroots technical summit dedicated to spreading awareness and knowledge about free / libre / open source software, hardware, and culture.
SeaGL strives to be welcoming, enjoyable, and informative for professional technologists, newcomers, enthusiasts, and all other users of free software, regardless of their background knowledge; providing a space to bridge these experiences and strengthen the free software movement through mentorship, collaboration, and community.
Following a question from someone about getting started with Matrix bots, I wrote a super quick getting started guide to writing a simple bot with opsdroid. I will probably integrate this into the opsdroid docs, or somewhere more accessible in the future, but for now here it is: https://gist.github.com/Cadair/2c64cb46fdce7499ff1c0bbb116ce29e
Being an innocent child, I asked:
are you aware of projects relying on opsdroid to make a Matrix bot? Other than half of the very famous TWIM bot 😇
I have written a few (https://github.com/Cadair/skill-motw, https://github.com/Cadair/skill-intersphinx, https://github.com/Cadair/skill-jitsi) I know that Oleg has also written a number, including many which interact with homeassistant.
I didn't know what I had started! We almost became This Week in Opsdroid! Fortunately the Matrix community is very active and helped us keeping sovereignty over the blog. Phew!
There are too many to list here, but I strongly encourage you to blend in with the opsdroid community. Not only are these kind people, they are also very active and develop fancy and useful bots!
While still disabled by default, we're beginning to land code in Synapse to support threaded discussions in Matrix. In particular, the m.thread event relationship defined in MSC3440 is now implemented behind a configuration flag.
There are still many open questions here, and MSC3440 has not yet been approved to merge into the Matrix spec, but it's a start. Threading is essential to Matrix's continued growth and adoption, and we're excited to provide server-side support to this effort.
Bug Fixes and Refactors
This release is mostly comprised of bug fixes and improvements to static typing. Of note:
The export-data admin command now works and is tested in CI. This command helps server administrators respond to GDPR Subject Access Requests.
A weeks-long effort to refactor how Synapse validates the auth_events field of incoming PDUs has concluded, resolving a few corner cases which could incorrectly allow events into the room state which should instead be rejected.
Further static type hints have been added to Synapse, improving our precise type coverage to 85% of all lines. Currently, 13% of the Python files in synapse/ are skipped during mypy runs. We'd like to cut this in half by the end of the year, as well as increasing overall precise coverage in the codebase.
Sydent 2.5.0
This week also saw the release of Sydent 2.5.0, the reference implementation of a Matrix Identity Server. In addition to fixing bugs, Sydent 2.5.0 passes mypy --strict, uses Jinja2 for templates, and supports the room_type field from MSC3288 to better differentiate between invitations to rooms and invitations to Spaces.
Everything Else
The Admin API to create or modify accounts now accepts a user_type field, allowing for accounts to be set as belonging to bots or support staff.
Password auth providers can now be implemented using the new pluggable extension module APIs.
Please see the Synapse Release Notes for a complete list of changes in this release.
Synapse is a Free and Open Source Software project, and we'd like to extend our thanks to everyone who contributed to this release, including aaronraimist, AndrewFerr, dklimpel, and Legogris.
This week my guest is Element's Kat who conducted various Community Testing Sessions. If you want to learn more about it, why Kat is conducting them, and how you can help, have a look at it!
TL;DR: Europe is close to officially regulating big corporations and forcing them to open their silos via the Digital Markets Act. Element joined forces with other companies to make sure that the members of the European Commission about to vote on the regulation knew how important it is.
Matrix was created to break the silos, as an interoperable layer for all communications. We wanted to ensure people had the ability to choose where their data is stored, and had the choice of interface to access them. This was also a good way to enable better competition and innovation in the communication space, forcing the players to compete on value add. But whilst Matrix is already a 39M users open network in itself, it could be even better if the bigger silos decided to open up and participate into it.
In December last year, as part of the Digital Markets Act (DMA), the European Commission proposed to update the competition law for digital markets with rules mandating ‘gatekeeping’ platforms to open up, in order to enable innovation and data sovereignty.
This move is so aligned with Matrix and Element’s mission, that Element joined over twenty other organisations to launch the Coalition for Competitive Digital Markets on Tuesday the 26th. The coalition supports stronger rules for large online platforms in the DMA, calling for amendments to it to make sure that major services offer a public API to interact with their core services and to ban the pre-installation and default setting of core platform apps.
Matrix is a perfect example of how technology can thrive through openness and collaboration. We will bring our experiences of using federated and decentralised technologies to MEPs and Council members, in the hope for a regulatory framework which supports European innovation, competition and interoperability.
Here's your weekly spec update! The heart of Matrix is the specification - and this is modified by Matrix Spec Change (MSC) proposals. Learn more about how the process works at https://spec.matrix.org/unstable/proposals.
Matrix 1.1 is just around the corner! The team continues to work hard on getting the infrastructure surrounding the new release process set up, as well as reviewing MSCs, including the ones you can see above!
The current speculative release date is in about a week or two. Keep your eyes peeled!
This is one of those MSCs that will likely be sorely needed at some point in time, but currently hasn't gotten a lot of attention. I can think of all sorts of usecases for this (on top of the many that the MSC puts forwards). Go ahead and give it a read!
Over the weekend, one of the servers backing matrix.org suffered a hardware failure, leading to around half an hour of downtime as we failed over to another system. Remediating and monitoring that has been the focus of much of the Synapse team over the past week.
We also shipped a release candidate for Synapse 1.46, due for formal release next week. Notably, we believe this solves a performance regression which was introduced in Synapse 1.44. If you've been experiencing long hangs or other periods of unresponsiveness, please try upgrading to 1.46.0rc1 and let us know if it resolves your issues.
The Synapse team has also decided to aim for building a rough prototype of MSC2775: Lazy loading over federation by the end of the year, in hopes that this will pave the way to extremely fast joins of large rooms.
We're also continuing to work on static type checking in our projects: As of today, Sydent passes mypy --strict! We hope to write about what we've learned in the near future.
Here's an addendum about Sydent. We can see the results of our efforts visually, thanks to mypy's reporting options. Two different metrics show increasing coverage since our typing efforts began in the summer. The grey strip shows the last fortnight's sprint in particular.
One of the shortfalls facing XMPP users of the Bifröst Matrix bridge is the lack of support for Message Archive Management (MAM), which allows XMPP users to retrieve messages sent when they are offline, when they come online next time. Sunday Nkwuda and Olatunji Ajayi, with help from Pirate Praveen, are planning on writing support for MAM in XMPP.js (the XMPP library that Bifrost uses), and then adding MAM support to Bifröst. Since this requires some dedicated full-time effort, they are fundraising to support this project at https://opencollective.com/mam-plugin-for-xmppjs. Many thanks to the XMPP Standards Foundation for serving as fiscal host for the fundraising.
Free form AUTOCMD after joining a channel, per channel
Automatic rejoin on invite or kick (invite enabled by default)
AVATAR network room command to set avatars for IRC users (admin only)
WHOIS room command for PMs, PART for channels
Add ROOM command in network room to run channel room commands safely
Move PLUMBCFG stuff under ROOM command
A smaller release this time around which consists mostly of QoL stuff to make things a little easier. IdleRPG users can now rejoice and login to your character automatically with channel AUTOCMD when joining on reconnect as what's more important: running a bridge to engage in deep conversations or sit in a channel doing nothing?
Plumb users need to use the new generic ROOM command in network rooms to configure plumbs now that PLUMBCFG has been removed.
Several bug fix releases this week, and hope to get out another release tonight with support for dehydrated devices, which should prevent you from missing any messages while you are logged out of all your devices.
Our next community testing session on Web will be at 16:30 - 18:00 BST on Wednesday, 3rd November. We will be focusing on voice messages and calls. Join us in #element-community-testing:matrix.org to find out more and help out.
Added sourcemap support to the sentry telemetry sent with rageshakes, and added support for opt-in automatic sentry reporting when errors occur as a labs flag
We’ve been working on making notifications and matrix.to work correctly with threads in Element Web. Our MSC continues to progress, receiving feedback from the Spec Core Team this week, and we’ve started to plan how to test the feature.
We’re also exploring more intuitive ways to lay out and customise Spaces on the web.
Element Android 1.3.6 is on its way to the PlayStore but Google is taking looooong time to review. Release candidate 1.3.7 is scheduled to next Wednesday
Replacing Rx by coroutines Flow is nearly there, and will be merged on develop after the release 1.3.7.
Besides that we have started to work on Threads and on Polls.
Some changes have been made on the SDK API to suit SDK users regarding the AuthenticationService. They will be available in SDK 1.3.7 (next Wednesday).
And as always, we work to improve the performance of the application and of the SDK, as well as fixing bugs here and there.
We added two (opt-in) lab options:
Automatic bug report
Structured login / open telemetry (offline)
On the crypto side, we're making progress on Android Rust SDK, room shields and key backups
Last week we had a successful and productive community testing session on Android where we found many issues, including 21 defects, which are already being addressed by product, designers and developers.
Fix autofocus in msg input upon clicking emojis in emojiboard.
Find more about Cinny at https://cinny.in/
Join our channel at: #cinny:matrix.org
Github: https://github.com/ajbura/cinny
Twitter: https://twitter.com/@cinnyapp
Hello again! Halcyon is a Matrix bot library created with the intention of being easy to install and use.
This release brought some new non-breaking features:
Added markdown package as a dependency, used in formatted messages
Added support for the following message types: TEXT, EMOTE, NOTICE, IMAGE, FILE, AUDIO, VIDEO
Added support for downloading and uploading MXCs (matrix media)
Fixed a reported import issue when trying to use the CLI (Thanks @Josh)
Added more documentation and a new example in usage.md
More info at on the project at https://github.com/WesR/Halcyon
Chat with us over in #halcyon:blackline.xyz
Tobias Fella and I are going to give an introductory-level talk at Qt World Summit '21 next Wednesday (3-Nov) about data encryption in Qt applications, with me giving my usual general bla-bla high-level overview and then Tobias talking about really specific stuff like Base64 using std::variant and a bunch of Qt classes to make your encryption-using code less of spaghetti and more of good-looking modern C++. The talk will be at 15:30 CET/CDT (once for EMEA/APAC and once for Americas), Platform track. Advance registration is necessary. In fact, the talks are pre-recorded but both of us will be online during the session and shortly after so please come around to make sure The Qt Company gets the message and considers adding Quotient as another module to Qt (ok, that's just a dream for now but who knows...)
Heads up for those in Berlin. You're welcome to join us Tuesday, 2nd Nov at 7:00 PM chatting about Matrix development and hosting. We're going to meet in person at c-base. In compliance with the hackerspace's house rules this is a strict 2G event.
If possible, join our #matrix-berlin:matrix.org room.
I work on Collabs, a library for making decentralized collaborative apps, and we have some demos that run on Matrix! E.g., in a widget-capable client like Element, send the message:
to add a collaborative text editor widget. (Fair warning: loading and saving is flaky, so don't type anything you can't afford to lose.)
All collaboration happens using messages sent to the Matrix room. So, you're not dependent on any external service provider, and the document is end-to-end encrypted if your room is.
We have more demos here, with instructions on how to use them in a widget. You can follow those same instructions to run your own Collabs apps (Getting Started Guide), without needing to host any servers yourself!
If you want to learn more about Collabs and see an app running on Matrix, check out my Strange Loop talk: https://www.youtube.com/watch?v=Exr0iY_D-vw&t=3s. The library is in an alpha state and not yet ready for production use, but we'll be improving it steadily over the next year.
In FediJam feedback we were asked to explore federated gaming.
So, for my third FediJam I made a dead-simple prototype of a game using Matrix not for chat, but for multiplayer.
Behold, the Federated Triangle of Doom.
Game creates a chat. Any message with 'boom' posted into there spawns the Red Triangle of malice.
Yes, literally just it.
Behind the scenes, it uses Godot Engine, with godot-python to make it python-capable.
To connect via Matrix, it leverages matrix-nio python module.
As a server, any Matrix server without captcha/email for registration will do!
For simple local tests, I included conduit with the Linux release.
Element is on the hunt for more VoIP and VR developers to help us build next-gen native Matrix video conferencing and VR/metaverse experiences on top of Matrix! If you believe the future of the metaverse should be free/libre, standards-based, open and equitable, come join the resistance! https://apply.workable.com/elementio/j/25BB112FBD/
Julian Sparber from the Fractal team was here with us this week, to tell us about the Fractal Next initiative: a rewrite of the app that leverages new technology. You can talk to the Fractal community by joining their room at #fractal:gnome.org.
I've been working on a brand new /sync mechanism, dubbed sync v3, for the past few months. This is a complete overhaul of how clients live-stream sync data from homeservers, and fixes numerous pain points with /sync including:
Syncs taking minutes to complete.
Syncs sending back vast quantities of data (e.g read receipts for every room you're in).
Currently, this new API is designed around the idea of having dynamic sliding windows around a sorted room list for a user's account. The room list can be sorted in different ways and only rooms in the window are returned to the client. The room list can also be filtered down based on various criteria (e.g only rooms in these spaces, with events newer than this timestamp, etc) and a subset of current room state can be requested with each returned room.
All joined rooms on user's account
Q W E R T Y U I O P L K J H G F D S A Z X C V B N M
\ /
\ /
\ Subset of rooms matched by filters /
Q W E R T Y U I O P L K J H G F D S A Z X C V
|
A C D E F G H I J K L O P Q R S T U V W X Y Z Rooms sorted by_name (or by_recency, etc)
|_______|
|
A C D E F first 5 rooms requested
This list is kept sorted on the server and deltas are communicated to the client as a series of delta operations (e.g INSERT,DELETE,UPDATE). Multiple non-overlapping windows can be requested and the windows can also change their size dynamically.
This work is still currently in the pre-MSC stage as it's a huge undertaking, but an early, runnable, implementation lives at the sync-v3 repo. This works by setting up a proxy sync server which will make /sync requests to a target homeserver, and storing the data in a useful form for sync v3 queries. It then exposes the sync v3 API which can then actually show you real data from your account.
+---------+ +--------------+ +----------------+
| sync v3 | <-- sync v3 API --> | sync v3 | <--- normal /sync requests ---> | Any Homeserver |
| client | | proxy server | | Impl |
+---------+ +--------------+ +----------------+
The proxy server also hosts a barebones web client which talks sync v3 at http://localhost:8008/client/ - to get started just point the proxy server to any HS and then paste in your access token to the web client. The web client is just a test jig so only has a read-only view of your account and lacks any scrollback, only showing live data. IMPORTANT: generate a new access token before using this client. Do NOT use the token from an existing client (e.g Element web) or else you risk being unable to decrypt E2EE messages.
Intrepid people are encouraged to run a server and see how fast sync on their own account could possibly be in the future. Note that the first sync will be slow as it has to fetch all the user's data via v2 /sync.
I know it’s going to be a while before this finally gets into our hands as users, but I’m looking forward to it!
Here's your weekly spec update! The heart of Matrix is the specification - and this is modified by Matrix Spec Change (MSC) proposals. Learn more about how the process works at https://matrix.org/docs/spec/proposals.
FCP is currently paused while everything that isn't currently implemented in the wild is removed for a future MSC.
New MSCs:
There were no new MSCs this week.
Spec Updates
This week the Spec Core Team has been marching ahead with getting the matrix-doc repository ready to start building new spec releases. Big props to Rich, Travis, Alexandre and others for charging ahead with it!
Otherwise, the Spec Core Team has been busy reviewing MSCs as can be seen from the above list of MSCs that have entered FCP this week. Particularly exciting is to see MSC2674 (event relationships, aka the stuff that powers message edits and reactions) coming close to merging! It's been a long time coming. Thanks to Bruno and everyone else who's reviewed it so far.
MSC3214 looks like an MSC that's been somewhat abandoned by everyone involved... But since there are already implementations out in the wild, it may be a simple one to get through!
This week we have spent some time optimising the Pinecone overlay routing protocol. We have added cryptographic signatures to the bootstrap and path setup process for stronger verification and have also reduced the amount of protocol maintenance traffic.
We are also finally starting to document how Pinecone works — you can find these living documents on the Pinecone repository wiki and many new comments have been added throughout the source code too.
We have also released build 88 of the P2P demo today for iOS and Android. These demos are forks of the Element app with an embedded Dendrite homeserver and Pinecone-based federation. Users can communicate with each other locally on the same Wi-Fi network, direct between devices in close proximity using Bluetooth Low Energy and globally by using internet peers!
Dept of Servers 🏢
Synapse
Synapse is a popular homeserver written in Python.
This is the first release to support oEmbed autodiscovery, leading to richer link previews for many sites. See the release announcement for an example.
We also believe that we've finally fixed stuck messages! A race condition would occasionally prevent your sent events from syncing back down to all of your clients. This caused messages to look like they were stuck at the bottom of the room, waiting to finish sending, even though other users would receive and see them normally.
Please shout loudly in this GitHub issue if you see a stuck message on a server running 1.45 or later. 😉
We've merged a further 9 pull requests to Sydent, making even more modules pass mypy --strict. We're not quite all the way there, but we're close! Most of the "leaf" modules are done, with the bulk of the remaining work to be done is on sydent.http. This is trickier because it's more exposed to the machinery of Twisted, but we'll get there.
We're already much more confident working in Sydent, and hope to apply these lessons to Synapse soon.
(Well, one lesson that won't apply is that type hints are especially valuable in codebases you don't touch as often... but we have no intention of slowing down on Synapse any time soon!)
This week too sees some Helm Chart updates, taking matrix-synapse first to 1.45.0 and then to .1 - as well as improving support for using existing secrets in a new deploy.
Heisenbridge is a bouncer-style Matrix IRC bridge.
Release v1.3.0 🥳
SOCKS proxy support
CertFP authentication support
WHOIS command and reply formatting
Basic CTCP support (optional, default off)
Compatibility/workaround for Dendrite and Conduit registration configuration
README has been improved
Big ticket items this time around for privacy concerned users are connectivity improvements when using Tor/VPN with built-in SOCKS proxy support and CertFP authentication. Some WHOIS and CTCP has also been thrown in the mix for good measure!
Lately there has been quite a lot of people who are testing Dendrite or Conduit which has led to recurring support issues around some limitations they currently have with appservices. Heisenbridge now supports using and generating a "compatible" registration file (--generate-compat) for them that shouldn't be needed in the long run. I hope to drop this feature as soon as possible! 🍻
While Heisenbridge somewhat intentionally has very little documentation due to not having a refined UX yet the README has been updated with some frequently asked questions about installing and configuring it to the point you can talk with the appservice bot. A self-contained docker-compose setup has also been added to the repository to help testing it without installing anything on your system.
Last Saturday we had an improvised NeoChat mini development sprint in a small hotel room in Berlin in the occasion of the 25th anniversary of KDE. In a good KDE tradition, Carl spent this time on improving NeoChat settings. He ported both the NeoChat general settings and the specific room settings to the new Kirigami.CategorizedSetting component. Tobias fixed a lot of papercuts and now the power level should be fetched correctly, we show the number of joined users instead of joined+invited users in the room information pane, the user search is now case insensitive. Nicolas focused on fixing our Android build by making the spellchecking feature compile on Android.
Aside from the mini-sprint, we also made a few more improvements during the week. Tobias fixed the flicking of the timeline on mobile and Carl made it possible for the user to resize the room information drawer.
Nheko
Nheko is a desktop client using Qt and C++17. It supports E2EE and intends to be full featured and nice to look at
Thulinma added device management to Nheko. You can now also view your devices, that don't support encryption, delete devices, rename them and see their ip address. Additionally there were a few refactorings and code improvements. Most notably blurhashes should now decode twice as fast. Additionally there were a few small fixes to displaying hidden space rooms (we were checking the wrong power level, so in some cases fewer rooms got shown in a space, if they only had a parent set). That's all, was a busy week with lots of lasers!
Element
Updates from the teams:
Web
Finalised our team goals of reducing prioritised defects, shipping threads MVP, better release automation and getting posthog analytics live!
Working on clarifying the threads MSC in response to feedback
Working on user prompts for analytics
Fixed bugs in RTL text and code block rendering
Released Element Web 1.9.3RC
Spaces
On iOS, we’ve been implementing support for creating Spaces, implementing the screens using SwiftUI.
We’re also making good progress on iterating on the info architecture of the app, to make Spaces more intuitive. Expect more things soon!
iOS
Publish release 1.6.6 containing matrix.to display behavior improvements on timeline and other improvements made this week.
Improve matrix.to redirection in timeline, pushing views instead of replacing current ones
Improve settings UI
Allow images to be pasted from Safari rather than their URL
Implement message forwarding
Improve wellKnown parsing
Fix bugs and crashes
Android
Release candidate 1.3.4 has been prepared Wednesday and tested by the community during a testing session in the afternoon. It adds presence support for DM, and Android Auto support has been restored. You can find out more about the next testing session at #element-community-testing:matrix.org
We are preparing a version 1.3.5 which fixes small regressions and a bug in the Room settings.
Migration to Hilt (Dependency Injection library) is nearly there, PR is in review.
Notifications management has been rework, a bunch of issues has been fixed, this will be for release 1.3.6
We have started to work on Threads
Crypto Team:
UISI Hunting: Investigations and Auto UISIs reports lab option, UI improvement work started
Web: Device Trust view improvement
Next gen
Corroded-Android: Backup support
Messaging Layer Security
Fractal
Not only is Fractal’s Julian featured in this week’s Matrix Live, he also has been very active in the month since our previous report. I won’t list here the very long list of merged merge requests, but the most noteworthy changes he brought are:
Hey, TWIM. Did you ever face a situation where you need to run any command prefixed with time command-name and get a ping in matrix room when it's done? No? Well, doesn't matter, now you can do that with one small command!
That thing called time-to-matrix (binary name: ttm) it works like time command-name and sends results to matrix room. Yes, pretty simple. Now you can use it 😀
Element One is here! We’re giving TWIM a sneak preview of the new Matrix hosting service from Element in advance of the official launch next week. Element One* is a consumer-focused hosting option which gives you an individual Matrix account on our shiny new one.ems.host homeserver (with a matching Element Web at one.element.io). However, in addition to being a fast, snappy Matrix account, it also comes with unlimited personal bridging to Whatsapp, Signal and Telegram thanks to mautrix-whatsapp/signal/telegram!
Hopefully this is good news for those of you who have chats with friends and family on these other platforms, as you can now chat with them in one place from Element without having to mess around having to run your own bridge - and accounts cost just $5 per month.
Right now this is a (relatively) private / limited beta intended for the TWIM and EMS inner circle, and we'll be announcing more generally in the next few days / weeks. So, if you are interested, please check it out and sign up at https://ems.element.io/element-one :)
Good news - Miounne got new release with a lot of small neat changes (and even some features, yes-yes) and binary releases for major platforms and architectures
Good/bad news - e2e integration was removed, because I failed to implement to "well, usable" state 😭
So, go check out releases, changelog and don't forget to pull new docker images.
You asking what's that about? Mother Miounne is "backoffice" of etke.cc service.
I'm too sad to add good jokes and brief descriptions, so just join the #miounne:etke.cc room to find more
SeaGL (Seattle GNU/Linux Conference) will be held online November 5–6, powered by Matrix. Join our free—as in freedom and tea—grassroots technical summit. No registration required! This year's theme is "on cloud nine", in recognition of our ninth year.
Keynote speakers are:
Marie Nordin
Christine and Morgan Lemmer-Webber
Elana Hashman
Cory Doctorow
We have over three dozen other great talks on free and open source software, hardware, and culture. Plus some fun social activities. Volunteers are still needed if you want to lend a wing!
Hi everyone! Did you ever feel lost in the Matrix world? The room directory is big, but it's still hard to find something you like. Or are you a room moderator, but there is not much activity in your room because it doesn't have enough users?
This is why I want to share rooms (or spaces) I find interesting.
Synapse 1.45.1 is out now! Python 3.10 and PostgreSQL 14 are now tested and supported by Synapse. Support for Python 3.6 and PostgreSQL 9.6 will be removed by the end of the year.
Note: This release may require changes to how media storage providers access your homeserver's configuration. See the Upgrade Notes for more information.
Note: Synapse 1.45.0 was released yesterday and changed how Synapse's monthly active user limits were calculated. Today's release of 1.45.1 reverts that change, but is otherwise identical to 1.45.0.
Support for oEmbed Autodiscovery
Synapse can now automatically discover rich metadata when generating previews of links to sites which support oEmbed.
Before:
After:
Note that URL previews are generated server-side, and thus generally disabled in encrypted rooms to avoid leaking information about message content to your homeserver. You may need to adjust the room's settings to see the new oEmbed previews.
Fixing Stuck Messages
This release of Synapse fixes a race condition which would occasionally prevent your sent events from syncing back down to all of your clients. This caused messages to look like they were stuck at the bottom of the room, waiting to finish sending, even though other users would receive and see them normally.
Improved Privacy for Per-Room Nicknames
Matrix allows users to set their display names to be different things in different rooms. For example, you might use an alias in public rooms, but your real name in rooms shared with friends and family.
To make it easy to initiate conversations with people, each homeserver maintains a user directory with the Matrix ID, display name, and avatar of the users it sees. Previously, this directory would be updated with the most recent profile metadata that Synapse had seen for a user, even if it was only changed in a single room.
As of 1.45, Synapse only uses includes the default display name of local users in its user directory, ignoring room-specific nicknames or avatars. (#5677).
Internals
This release includes numerous fixes and improvements to Synapse's internals.
We've added countless static type annotations to Synapse (and related projects like Sydent), giving us greater confidence in its correctness and reducing maintenance costs. Several modules newly have all of their definitions typed, allowing us to require and enforce complete type coverage for all future edits therein.
This release includes meaningful fixes and improvements to our OpenTracing and logging machinery, helping us better catch and eliminate bugs in Synapse. This work ultimately reduced matrix.org's Sentry event volume by an order of magnitude.
Magic accessor methods have been removed from Synapse's Config class. Previously, Synapse would interpret references like config.send_federation by attempting to guess a reasonable full path, like config.worker.send_federation. As of Synapse 1.45, the full path must be specified directly. This prevents errors where values could be drawn from unexpected or incorrect sections of the server's configuration.
Everything Else
We'd like to extend a special thanks to Fizzadar from Beeper for improving Synapse's update_synapse_database script (#10954) to allow schema changes to occur while Synapse is running. This is a great step toward reducing the downtime associated with upgrades.
These are just the highlights; please see the Upgrade Notes and Release Notes for a complete list of changes in this release.
Synapse is a Free and Open Source Software project, and we'd like to extend our thanks to everyone who contributed to this release, including AndrewFerr, dklimpel, Fizzadar, lukaslihotzki, and maxkratz.
Here's your weekly spec update! The heart of Matrix is the specification - and this is modified by Matrix Spec Change (MSC) proposals. Learn more about how the process works at https://spec.matrix.org/unstable/proposals.
Additionally, take a look at the Client-Server API, where all current APIs have been bumped from /r0/... to /v3/.... See this bit of MSC2844 for the rationale behind this change.
The pieces are starting to fall into place for the upcoming Matrix 1.1 spec release!
This is a concept that was used in the Spaces MSCs to allow servers and clients to denote a room as a Space vs. a typical Matrix room (and set up their UI appropriately). Only the type for a Space has been specified so far, but there are many different applications room types could serve; for example profile rooms!
This MSC aims to create a general concept for a typed room, whereas future MSCs would identify those types and their meaning.
Happy Friday! The Synapse team has been sorting out what we hope to accomplish before the end of the year, and it's looking like our main focus will remain research and experimentation towards making large room joins go very fast. We don't yet see an incremental path from where we are now to where we want to end up, so progress will likely come as a few discrete milestones, rather than a gradual acceleration over time.
Aside from room joins, we expect to spend a good bit of time time on bug fixes and preventative maintenance. For example, we hope to get Sydent passing mypy --strict by this time next week. Doing so would have prevented recent production issues, including one which caused phone number verification to silently fail.
Preparations for releasing 1.45.0 continue apace; I'm looking forward to telling you about it next week!
let's bridge everything! Have you ever wished you could make and receive phone calls with matrix?
The beginnings of a matrix <-> SIP bridge are done :) If you can imagine to live with bugs or even help with the development yourself, feel free to give it a try! :)
https://github.com/alangecker/matrix-appservice-pstn
This release fixes several bugs and makes E2EE enabled by default.
feat: Enable E2EE by default for new rooms
feat: Display all private rooms without encryption as red
feat: New design for bootstrap
feat: New design for emoji verification
feat: Display own MXID in the settings
feat: More finetuning for font sizes
chore: Updated translations (Thanks to all translators!)
fix: App crash on logout
fix: Temporary disable sign-up for matrix.org (Currently gives "500: Internal Server Error" while FluffyChat should send the same requests like Element)
fix: Implement Roboto font to fix font issues on Linux Desktop and mobile
fix: QR Code scanning
Nheko
Nheko is a desktop client using Qt and C++17. It supports E2EE and intends to be full featured and nice to look at
Making steps towards stable E2EE, we have now implemented bootstrapping cross-signing on a new account. This allowed me to finally enable cross-signing on my Nico (nheko.im) account (which I intended to only use with Nheko). Thulinma also added a way to refresh the current device list as well as highlighting your current device in the device list, which should make it easier to recover, if a misscommunitcation with the server lead to an outdated device list. Meanwhile LorenDB has been plugging away on converting more dialogs to Qml and we merged like 3 converted dialogs!
Element
Web
Released Element Web 1.9.2 featuring chat export, Spaces and E2EE improvements, updates to Electron 13.5.1
More progress on the client side of threads, published an MSC and started work on Synapse support
Fixed some bugs in registration, Windows icons, phone number confirmation
Delight team
On iOS, we’ve been implementing support for pagination of the Space Summary API, to support larger spaces, as well as improving performance. We’ve also started implementing support for creating Spaces.
We’re also exploring larger changes to make to the layouts of Element on all platforms to surface simpler and more intuitive navigation all round and tackle some of the feedback we couldn’t get to during the Spaces beta. Watch this space!
iOS
We failed to publish the app on the app store this week because of observed crashes. RC 1.6.5 is available on TestFlight. We should be able to release on the App Store on Monday
User auto completion has landed on develop
Message forward is in the horizon
Settings got lipstick by reviewing headers and footers
It is now possible to copy and paste an image
Dept of Widgets 🧩
New Widgets: Spanners!
Can you believe we didn’t have a dept. of Widgets yet? We do now!
Hi folks! This week I took a bit of time out of my day to create a new widget for Matrix. We have the concept of "Spanners" at Matrix where one person wants to take control of a resource, and they will say "I am taking the spanner". This prevents other people (verbally at least) from trying to use the resource at the same time.
For years we have been getting by with sending messages into Matrix, but no more! For now anyone can use the Spanner widget to set up a similar system in their room.
You can try this out now with /addwidget https://half-shot.uk/spanner?spannerName=YourSpannerName (although no promises about the uptime of half-shot.uk)
The repo can be found https://github.com/Half-Shot/matrix-spanner-widget.
You will need to ensure that all users who plan to use it must be able to send state events of type uk.half-shot.spanner. Otherwise, it should be ready to go!
Dept of Ops 🛠
Rager
I heard CLI tools are all the rage, so here is one!
Rager is a CLI tool for matrix client developers to easily view and parse rageshake logs. It handles syncing (with many parameters to sync only the logs you want), searching through downloaded logs, and paging through any log on your device (with peudo-syntax highlighting to make it easier to understand), right from the comfort of your terminal. It has been in development for a few months (being used/tested by devs at beeper.com) but I feel like it's now in a good enough state to release for everyone else to use. If it's something that may help you, try it out and let me know if it works for you!
We believe that having your own homeserver should be easy and require little technical knowledge. That's why we launched Hingst, a privacy focused Matrix hosting service which takes care of (almost) everything for you.
Our goal is that our users spend no time whatsoever on maintaining the actual server so they can focus on what matters - chatting with friends and colleagues.
All servers are located in Sweden and of course they run entirely on renewable energy!
That's all for now. Thanks for reading :)
If getting your own homeserver was hinging on the availability of such a service, you don’t have an excuse anymore!
First annual Finnish Matrix community meet is to be held 20.11 in Tampere, Finland. Infos and registration at https://mobilizon.be/events/5c9ce49d-83de-41d1-b824-8950293d3fd1
Heads up for those in Berlin 🐻🇩🇪. You're welcome to join us Tuesday evening at 7:00 PM chatting about Matrix development and hosting. We're going to meet in person at c-base. In compliance with the hackerspace's house rules this is a strict 2G event.
Hi everyone! Did you ever feel lost in the Matrix world? The room directory is big, but it's still hard to find something you like. Or are you a room moderator, but there is not much activity in your room because it doesn't have enough users?
This is why I want to share rooms (or spaces) I find interesting.
This week, I had the great pleasure of chatting with a lovely bunch from Element about Spaces. This feature just got out of beta and they tell us all about it.
Hello TWIM, it's me, not-anoa, again with your weekly spec update. I unfortunately didn't get a chance to figure out how to run the tooling which generates fancy graphs, but it looks like you'll have a more qualified person next week 😇
This week we saw a couple new MSCs:
MSC3419 - Letting guests do more things (which is needed for things like full mesh conference calling)
MSC3429 - A room preview API. Turns out this is more or less a duplicate of another MSC, but has a different approach. Be sure to check both out 🙂
We've also been making progress on trying to get the spec release out by formally specifying all the new global version numbering and new endpoint versioning (see this and this), and have been experimenting with how room versions are represented by the spec.
From page 6 of the open MSCs there's MSC2723. The MSC is relatively small, but solves a fairly important problem with forwarded messages in Matrix: where were they forwarded from? From a glance, it has minor technical feedback and could do with an implementation (psst: client authors, this should be easy and fit well into a North American Long Weekend project 😇).
That's all I have for you today, but if I've missed something: sorry, and you should be in capable hands next week.
This week I have completely rewritten the Pinecone router with the aim of improving code quality and boosting performance. It now uses significantly less resources, making it much easier to simulate larger networks and using less battery power on mobile devices.
While there is still some work to be done to ensure the Pinecone network gracefully handles large-scale mobility events, protocol development is slowly progressing and we will soon be looking at how Matrix federation can be made more efficient by using routing intelligence from the overlay network.
If you have a passion for routing protocols and/or peer-to-peer systems, we're looking to hire engineers to work in the P2P Matrix space — get in touch by emailing your resume to us!
We also have our very own P2P Matrix room where you can follow along with the latest discussions and also occasionally pick up experimental iOS and Android builds of Element with an embedded P2P-enabled Dendrite homeserver.
We released Synapse 1.44 this week! The release was mainly focused on bug fixes and internal cleanups. In particular, we've made significant strides toward enforcing static type annotations across several more Synapse modules, we continue to remove "magic" code from our configuration handler, and we've landed significant refactors to both the user directory and federated event authentication. These changes pay off significant technical debt, reduce Synapse's maintenance cost, and ease future bug fixes.
But that's not all. This release also includes performance improvements around JSON serialization, new capabilities for extension modules, and better metrics around cache evictions. See the announcement for more details.
Hello one and all! We've got a minor-yet-MAJOR release candidate of the IRC bridge up for grabs. https://github.com/matrix-org/matrix-appservice-irc/releases/tag/0.32.0-rc1 is now based upon the 3.X matrix-appservice-bridge library, and so all the innards have been replaced with matrix-bot-sdk.
Please try it out, and the full release should be out next week.
Dept of Clients 📱
Element clients
Updates from the teams:
Web
Released Element Web RC 1.9.1 RC2, containing Spaces papercut fixes among other things
Threads is continuing to steadily progress
Cross signing bug fixes and usability improvements
Ran another community testing session, which has thrown up many suggestions for improvements to our onboarding experience
Delight team
Following the release of Spaces from Beta, we're fixing bugs and things which might have slipped through the cracks in the 1.0 release!
iOS
Released 1.6.1 with Spaces and URLs preview on the App Store.
RC1.6.2 is being built. This is mainly a maintenance release
The first maintenance release of Quaternion 0.0.95 is here, plugging more HTML injections (quite limited this time), with tweaks in positioning of the timeline to really return to the message you last read before closing the room, colourful display names in the timeline and not just in the user list, and one particular crash fix for a case when you made a typo in your password but then did it right - only to see Quaternion falling down to pieces. The release notes, together with self-contained binaries, are in a usual place, the Flathub repo has a new Flatpak too - just go and update!
Cinny v1.3.2 got released this week. It fixes unwanted "Password don't match" error on register page along with small improvements in Public room modal and message input (Thanks to Empty2k12). We also released v1.3.1 last week which fixed unnecessary CPU usages on idle and infinite spinner after login.
SchildiChat is a fork of Element that focuses on UI changes such as message bubbles and a unified chat list for both direct messages and groups, which is a more familiar approach to users of other popular instant messengers.
Both the Web/Desktop and the Android variant have received a couple of new features during the last weeks, so it's time again to share some of the highlights with you!
Web/Desktop
The Web/Desktop variant has received plenty of love for the last few weeks, so best to list some of the bigger changes:
Added a setting for the room list style: you can now choose between "roomy" with big avatars and two-line previews, "intermediate" for a smaller avatar and a single line preview, or "compact" to get a tiny avatar and the room preview in the same line of the room name
Added a setting how to color usernames in the chat: either uniform, based on the MXID, or based on the user's power level (Android has this feature for some time already, too!)
Added a setting to not open automatically the most recent chat when switching spaces
Improved theme settings: you can now choose custom themes independently for both light and dark mode
Workaround for switching between light and dark mode together with the system on Linux
The following screenshot showcases the compact room list and user name color mode based on PL:
So take that, IRC style lovers! 😁
Android
We added support for MSC2654 for server-reported unread counts. This means that we can now display the number of unread messages even for muted chats, if your homeserver supports it (synapse does)!
Hello! Halcyon is a Matrix bot library created with the intention of being easy to install and use. My goal is to lower the barrier of entry currently required to write and maintain a bot. I do not intend you to need a database, or have to juggle encryption keys.
With the initial release of the library we currently have:
A CLI tool for creating and revoking tokens
Hooks for new messages, message edits
Hooks for room invites, and leaving rooms
Sending messages as text, markdown, and HTML
Sending replies to messages
Ever improving documentation
Please check out the example in https://github.com/WesR/Halcyon and join us over in #halcyon:blackline.xyz
This week has seen two maintenance releases of libQuotient - just after 0.6.10 went out, fixing an issue with invites not showing up, another issue with URLs that have unescaped double-hashes (coincidentally, matrix.to URLs to some IRC channels are exactly those). Those URLs are not quite following the RFC but are accepted widely enough to warrant a fix, and a new release. Get 0.6.11 from here (or better, just checkout the respective Git tag).
A few weeks ago, I posted about the standupbot that I wrote to help assist with creating and sending standup posts to a room. I'm excited to announce today that you can choose to interact with the standupbot using threads! Instead of asking you about each part of the standup post individually, it creates threads for each of the parts that you can fill in. Your replies get added to the standup post like before, but it's a bit more interactive of a process. Here's a picture of how it looks (ignore all the display bugs due to Element).
If you are interested in learning more, the source code is here: https://sr.ht/~sumner/standupbot and you can join the development matrix room #standupbot-dev:sumnerevans.com
Dept of Events and Talks 🗣️
Running conferences on Matrix: A post-mortem of Ansible Contributor Summit
An (unofficial) Matrix server for the Newgrounds community.
Since I was pressed for time last week, I didn't go much in depth on how this server works. Here is a brief overview of its components:
https://app.ngmvs.one: A self-hosted Element Web with some theming tweaks, and the easiest way to visit the server.
https://matrix.ngmvs.one: the homeserver itself (Synapse). It has open registration, but only for Newgrounds Supporters (ie. patrons) via SSO logins with Newgrounds accounts. But Newgrounds doesn't have its own "Log in with Newgrounds" capability, so that leads me to the next component:
ng-saml-idp: middleware built on PySAML2 for using Newgrounds logins as a SAML2 Identity Provider. The code is tailored for my Synapse installation, but it can easily be tweaked to allow Newgrounds SSO logins to any SAML2-compatible service!
W-Bot: a Matrix appservice/bot built on mautrix-python that serves two purposes:
Creates Matrix rooms for Newgrounds movie/game submission pages (ie. maps #portal_view_<ID>:ngmvs.one to https://www.newgrounds.com/portal/view/<ID>), and adds a widget containing the submission itself to these rooms (but only if it's a movie--widgets for games will only have a link)
Restricts membership of any room it moderates to logged-in Newgrounds Supporters. Users without a :ngmvs.one account can authenticate to the bot via the "login" bot command, which lets them log into their Newgrounds account.
Point 2 also means that anyone on the Matrix network can make a room that's restricted to Newgrounds Supporters. Just make sure the bot has a high enough power level to kick users!
Public (and federation-accessible) instance is at @w-bot:ngmvs.one.
(PS: The "W" stands for "double"--because of its job of "copying" rooms into Matrix.)
MVSX: a Firefox extension that, when visiting a Newgrounds movie/game submission page, puts a button in the URL bar that sends you to the Matrix room created for that submission by W-Bot.
This week's updates:
W-Bot now has the inviteme command to invite the issuing user to the invite-only space room for logged-in Newgrounds Supporters. (Users should be invited there automatically, but this command is for people who leave / reject invites to that space & want to re-join it later.)
The MVSX extension has been updated to allow joining rooms with Matrix clients other than app.ngmvs.one. Right-clicking the URL bar icon now shows a drop-down list of alternative clients, and lets you set which one to use by default (on left-clicks of the icon).
Hi everyone! Did you ever feel lost in the Matrix world? The room directory is big, but it's still hard to find something you like. Or are you a room moderator, but there is not much activity in your room because it doesn't have enough users?
This is why I want to share rooms (or spaces) I find interesting.
"I‘m in plenty of retro computing and gaming related Discords and got annoyed that there was no such thing on Matrix. So, I created a Matrix Space for all things retro computing and gaming, hoping there might be enough like-minded people on Matrix to build a community. There should hopefully be rooms for everything now, although a few are still missing room icons - that will come over the next days. Here’s to hoping that the retro computing community can also thrive outside of Discord 😁."
