Hey guys!
Many things have been going on these past months, from the release of the new Volumio 3 plus the new pricing plan in December 2021 (we will go into it on a different topic in the next few days about it), we felt that we weren’t communicating enough about all things going on internally in Volumio, especially on the development side. Well, we want to change our current direction this 2022 and give a couple of minutes a week to tell you what is going on and to provide a bit more “insight” into what the development team is doing, as most of the time what they do is behind the curtain work, which is barely visible but that keeps the whole thing going.
Hope also that this will be a pleasant read also for the techies out there, and to provide some insights into what means to manage a project of the scale of Volumio. Read below what Michelangelo and the Volumio Development Team are working on now.
February
Week of February 21, 2022 – February 25, 2022
This week we have been focusing on scaling up some of our cloud systems, as since the launch of Volumio 3 we’ve seen a great spike in usage of Volumio and therefore the vital cloud parts of it.
Specifically, we increased the capacity of the infrastructure that makes it possible to access your devices from myvolumio.org, adding automatic ssl renewal to them, experimenting with a new brilliant piece of software (caddy server) and setting up 6 new servers in Asia, Europe and North America.
We also further expanded auto-checks on many new endpoints, following a shameful incident that made some of our repositories unavailable due to expired certificates.
We upgraded the musicbrainz mirror (thanks to the help of musicbrainz dev team themselves) and added a robust caching mechanism. This improves response times of metadata retrieval and also refined the algorithm to provide more accurate results and much more hits.
BTW, if you are curious to know how many servers\cloud servers\services are in Volumio’s architecture, head to:
It makes very apparent (to us in the first place) how complex is the architecture that we have to host to keep the project going. But this is just reflecting how technology has changed in the last years.
When we started Volumio we just needed one server (for downloads and OTA updates, and this server is still running), but now, a music player needs to interface itself with a myriad of third-party services to provide an adequate level of quality and integrations. This is also because in my opinion, a pure FOSS solution is not enough anymore in this day and age: you need to host a bunch of stuff and pay for it, to make it work.
We finally managed to track down a nasty bug that was affecting users with ISPs not using IPV6 or applying NAT, causing frequent stops in TIDAL, QOBUZ and everything that was streamed over HTTP2. This was due to a bug in curl (on buster) which made http2 requests fail under some conditions. We are currently doing QA on this fix, and if all goes well, next week we will release this update (for the joy of many users and our QA lead Davide).
Works on FusionDSP is progressing nicely, thanks to the effort of two brilliant members of the team and community, we are setting up a new open source component to make Fusion DSP finally work with Volumio’s AAMPP and we hope it can go public very soon.
We are also working on the new version of the Android\iOS app (which will be free). This new version will bring a super easy onboarding for the first time users, allowing them to auto update their devices and to connect to their wireless network in a very easy way.
This in our intention will make life easier for newcomers both of Volumio on Raspberry PI but also on Volumio products (that’s another thing we’re working hard on… stay tuned…).
Unfortunately, what we are really behind schedule is improving the plugins store submission procedure and testing. We have still a huge backlog of plugins to test and no one can really focus on it as the dev team is fully occupied. We are not holding it back however as we are actively looking for someone to take care of it as its first and only activity (if you are interested in being this person, drop me a message).
We also made some minor fixes to CD playback plugin, Tidal Connect, Metavolumio.
That’s it for now, see you next week.
March
Week of Feb 28, 2022 – March 4, 2022
First of all, all our thoughts are for Ukraine and ALL people affected by this tragedy, which is beyond imagination. The team is discussing on how we can send tangible support and we hope we can take action next week.
We decided not to take a political stance, nor to make any statement on the situation but rather focus on doing something tangible to help people in need.
Going back to frivolous stuff, our work this week has been focused on continuing to harden our cloud server infrastructure for remote connection, gradually moving all servers to the new and improved caddy configuration.
On Monday we released a new update which fixed quite some bugs, but unfortunately, the Tinkerboard image suffered from a temporary Github outage, which prevented it from installing some core packages like curl and the remote updater. We then issued an emergency update on Wednesday and our support team is helping those affected.
We are improving the build script in a way that the build will fail if this occurs again, avoiding us to promote releases with missing core-packages.
Two new open-source alsa modules have been released and integrated in the main images, allowing the finalization of the FusionDSP works (which seems very close) and a useful facility for plugin developers.
There has been also bug fixing done for Metadata display on premium plans (from the search function) and some work on CD Playback.
Development on the new apps is currently on halt, awaiting results from testing from Volumio’s moderators and devs.
That’s it for this week. I hope that the next post on this diary will be more cheerful because peace has been reached (although it seems unlikely, but that’s what we hope).
Week of March 7, 2022 – March 11, 2022
This week it has been almost all “behind the curtains” work for the team, boring but very important stuff…
First, there has been an incident when releasing version 3.233: a temporary github outage prevented the build system to download and install some critical components of Volumio image. Basically, the Tinkerboard platform ended up without some very important libraries (curl, flac) and the updater. Other platforms, like the PI, instead were built without the classic UI.
But you already knew about that.
The team work has been directed to debrief the incident and to take proactive measures against it happening again: we tweaked the build system in a way that it will fail the build if that should happen again (thanks Ash!) and add some more tests into our QA checklist for new releases (thanks Davide for the hard work there!).
We will gradually refactor the build system to report (and fail) every minor issue in the build process. Besides increasing trust in release repeatability, it will be a very important step in deploying an automated build system (on a CI) which we are considering for a long time.
In the last times, the complexity of the project has increased substantially, so we are at the point where we need to set up robust procedures (quality, development, support) and debrief very often.
On an organizational level, this is increasing a little bit of internal bureaucracy, but we reckon that it’s a very important task to carry out to ensure we guarantee high-quality standards for our users. The mantra that we are having is: never make the same mistake twice.
Then, works on the new iOS and Android app is progressing, although with some showstoppers: we are trying to solve some compatibility issues with old android versions with the new Wireless set up flows: it appears that older Android versions are not able to receive properly the list of wireless networks to connect to.
We continue monitoring the cloud infrastructure, following the big upgrade that happened in the past few weeks. The scaling of servers and architecture seems to hold up pretty well, and the increased load is handled beautifully.
Last, but not least, since the biggest (and most daunting) bugs of Volumio 3 have been fixed, we are now focusing on minor bug fixing and tweaks, and we have a new version under QA testing, hopefully to be released next week. It will include fixes for Ui, network-attached storage mounting, metadata discovery and more.
As you might notice, developing, maintaining and making Volumio grow, involves a lot of work that is not obvious nor visible to the general public, but it’s equally important to make it work.
Week of March 14, 2022 – March 18, 2022
This has been a good week here at Volumio HQ. The main reason for that is that we somehow feel that we’ve managed to turn around the “bad climate” that we had in this community lately and re-establish the joyful vibe that we strive to cultivate in this place of the net.
We reckon that the release of Volumio 3 hasn’t been without issues: many people were frustrated by a couple of nasty bugs (which we solved) and the lack of plugins in V3. This meant that the community was filled with angry comments by users (and the famous vocal unhappy minority), which were expressing frustration about those issues. This vibe not only made this a quite less interesting place, but had a huge toll on the team’s morale and contributor’s “attachment” to the project.
Thanks to the suggestions of the private moderators’ group we decided to act and put a lot more effort in communicating proactively with the community (what you read is an outcome of this) and the users who were experiencing issues.
It looks like it’s paying off. We can sense that the spirit of this community is getting back. But we are not relaxing, this challenging time was an occasion for us to learn how to do better, and this is what we want to continue to do.
One process which is being reshaped a lot is our technical support and customer help department. First, we wanted to have a better “dispatching” of requests to the proper person in the organization. We achieved so by restructuring the contact page in a more “directive” way, to forward requests to the appropriate person (shop requests\returns, billing, technical support, partnerships, distribution etc.)
Then we wanted to provide faster answers to the people requesting technical support.
By analyzing supports requests we found out that about half of them could be solved pretty quickly without the need of our support people, by just providing small written hints.
So we implemented a chat-bot with automatically matches users’ requests with our knowledge base (which we are filling on an ongoing basis) and provide immediate answers to known problems or situations.
This has proven to work well for some kinds of requests, but really it did not for others. Analyzing the flow and some feedback by our users this week, we found out that this system was not providing enough feedback on the required actions for some users (we required some further action from user before initiating the support session) which felt frustrated of not receiving support and contacted us on other channels (Facebook, forum, etc).
So, we are working to change this part, by changing the user contact flow in those kinds of situations. We expect to have this change live in about 10 days.
So, for those users which felt neglected in the past month, we acknowledge the problem and we are working to fix it, so the next time you need us you’ll find a better experience.
On the development side, work this week has been again on our cloud ecosystem. If you wonder why that is, that we spent all this time on cloud stuff and not on the “Volumio” side of things, here’s why.
Back in 2018, we took a very important architectural decision: move all the integration intelligence to the cloud, rather than on the device.
For example: our albumart retrieval server used to run on the device. It basically identified the artist, album and track and made some queries online to fetch and download the most appropriate albumart for a specific track. This worked well. Or at least it did until one of those online services shut down or change their API (this happens more frequently than you might think).
So, you have 2 ways to solve the issue:
- Update all the software you have on the field (in our case more than 430k installs). This means that many users will experience the problem until they update their software. It also means that spotting this problem will come only after you have some users’ reports.
- Move the logic that aggregates online content to the cloud and have the albumart facility in Volumio just rely on that.
We took the second step. This brings some additional advantages:
- We can understand quickly if something goes wrong, since we can set up automatic monitoring of all the services we use.
- When we spot a problem and fix it, the fix is instantaneous for all our users, without the need for an update.
- Doing some optimizations like caching and load balancing we can offer faster response times compared to an algorithm running on the device itself, and make in turn the whole experience better.
This comes also with some disadvantages:
- It costs money to run those services. If we use the “distributed” computing power of the devices, it does not cost anything. But if we need to set up, maintain and host such systems on a scale required for Volumio, this has a cost.
Basically, every time you look for an albumart, Volumio has to pay for it. Considering the service we provide for some of those endpoints is free, we have to be very clever to optimize our technology to not be overly expensive (in those cases, we don’t use AWS but a mix of Kubernetes + dedicated VPS with high microservices density). - In case it fails, it fails for everyone at the same time. So you have to put in place automatic recovery strategies and monitoring and ensure adequate redundancies.
Some examples of the services that use this architecture are:
- Oauth login for Spotify, QOBUZ, TIDAL (and possibly more like LastFm)
- Albumart retrieval
- Browsing of QOBUZ and TIDAL
- Plugins store
- Webradio selection
- Metadata and credits retrieval
And yes, what needed some love this week was the albumart provider. A change in an upstream API deteriorated the performance of retrieval of artist arts. We were able to fix it and deploy a fix without requiring anyone to update. And the best part of it, is that most of you didn’t notice.
We are continuing with this architectural pattern, especially when we enrich functionalities, which might explain why you see fewer commits on the Volumio backend regarding such aspects.
Some of the things we already worked on (we are waiting for frontend parts to support it) are live concerts notifications and lyrics.
That’s it for this week! Hope you got a better glimpse of how also the community can impact Volumio, both in positives and negatives ways. Remember that and keep on helping make this place a good portion of the net.
Week of March 21, 2022 – March 25, 2022
First of all, we managed to package together the update which we will release next week. It includes various fixes for multiroom, utf-8 characters and the longly awaited AAMMP update (which will allow us to release the amazing Fusion DSP plugin).
What held us up from releasing was a mysterious bug that affected multiroom, which we took a while to identify (turned out to be a very silly mistake, but very hidden and difficult to spot).
Then, finally, we managed to get the new Android and iOS app to a level where we are happy with: the new improved and simplified onboarding flow works well in all circumstances.
It took a lot of debugging, testing, swearing… You can’t imagine how tedious sometimes can be to design and implement a very simple process…
Next time you try a piece of technology that just works, consider that a developer could have lost all his hair to make it work properly 
Our plan is to release a beta test of the new app next week.
This week’s focus has also been a careful analysis of our current state (as a project and as a company) and strategic planning. As they say, “tactics without strategy is the noise before defeat”.
We held various team meetings to pivot our strategy in 2 main areas.
The first is the OEM long-term strategy and approach. As you might know, Volumio OEM is a service we provide to quite a lot of Hi-Fi companies around the world. We basically empower them to release Hi-Fi products using Volumio Technology and experience. We are proud that more than 30 Hi-Fi companies worldwide choose Volumio to power their audiophile products.
This has worked well both for us and them for the past years and Volumio is what it is today also thanks to the financial and design contribution of our partners.
We however think that we can do better and be more ambitious with the OEM Partner program.
We saw a huge increase in requests to become Volumio OEM Partners since the release of Volumio 3 and a slight change in what new partners require from our technology.
Compared to 3 or 4 years ago, Hi-Fi companies tend now to have in-house IT departments and are asking us for less direct involvement and more freedom when implementing our technology and a more SDK-like approach.
So, we are redesigning our OEM program very similar to what Android does. So yes, our plan is that Volumio will become the Android of the music technology world.
Of course, following our “co-design” approach, we are involving some of our best partners in the decision and design processes.
The other planning involved Hardware. We have ambitious plans and some really, really nice things that we are working on…
We can’t disclose anything at the moment, but, if I were you, I would consider visiting this year’s
HIGH END Munich trade show…
That’s it for this week!