That's not a whole lot, given the size of your nodes. Are you trying to optimize for earned fees? Or throughput? I guess a 95th percentile means that only 5% of your transactions are larger than 173k sat?
Discussion
Yes, the 95th percentile means that only 5% exceed the specified size, but day by day this figure fluctuates. Regarding fee optimization, I tried manually setting fees on all channels to approximately 800 ppm, and then progressively, over many days, lowering them and manually recording the number of successful payments, the total payment amounts, and so on in a spreadsheet. When you look at the results over a short period, it seems like you've found some optimal fee, but as more time passes, you realize that all these fluctuations have some other internal nature.
I tried to think about how optimization could be done, but it seems to me that this task is impossible because full optimization would require knowing the complete network graph with all node balances, which is fundamentally impossible. Moreover, this situation is constantly changing. Without knowing the balances of all nodes at a given moment, it's impossible to accurately predict which node will be optimal at any given time, which fee will be optimal, and so on.
Furthermore, payment paths and fees are set by the sender. The sender draws their information from the network graph, which always differs for each observer due to the slow propagation of gossip information.
Simple example. Previously, I tried to balance channels using fees. I had an algorithm where if a large portion of the channel's funds was on my side, I set fees low until the balance shifted more toward the remote side. Once, I encountered a situation where I couldn’t send a payment to a seller through my nodes. I investigated and found that my wallet was receiving an error stating that the fee on one of my nodes didn’t match the fee the wallet obtained from the network graph. In other words, the node had a higher fee than what my wallet had in its data. I continued to investigate and discovered that the node had raised the fee several hours ago and reported it to the network via the Gossip protocol, but my wallet, which was always connected to the network at that time, did not receive this update and still had the old information.
I don’t remember all the exact details of that case now. I know that in the packet returned to the sender in case of an error, the new fees present on the node at that moment are included. But for some reason, my wallet didn’t set the new fee and didn’t resend through that node. I don’t recall why that happened. However, while studying that case, I concluded that balancing channels using fees has significant drawbacks because the fee percentage changes over time, causing many payments that could have gone through to fail. Subsequently, I at some point made the fees on channels fixed.
Later, after some time, I had the thought that balancing using fees probably has a logical flaw. Because when we have a large balance on our side, effectively locking up a large portion of our funds, logic suggests that this is an ideal scenario to earn fees. But channel balancing pushes us to use minimal fees in such cases. And channel balancing also pushes us to raise fees when liquidity in the channel is running low. From a logical standpoint, I think it should be the opposite. From a profitability standpoint, that is. A node should earn fees specifically when it has a balance in the channel and not earn them when it doesn’t.
As a result of all my thoughts and experiments, I've concluded that I can't find any optimal way to effectively earn money and maintain a high flow. It turns out that when you increase the commission, at some point the number of transactions drops significantly, but you earn more, and this happens not regularly but in bursts. It's very difficult to determine where you're right and where you're not. Therefore, I got fed up with all of this, abandoned it, and set fixed commissions on the channels. Initially, I set them to zero entirely, and the flow increased significantly. Now, if memory serves me right, my commission is 10 ppm, and about the same flow is occurring as when commissions were completely zero, but now I'm earning around $5-10 per day. It's possible to earn more, of course, but I just got tired of constantly changing the code and adjusting commissions.
Of course, when talking about earnings, it's important to keep in mind that I'm simply taking the numbers from the commissions. But you have to consider that every day some channels close randomly or at the request of the remote party, some channels get severed due to protocol mismatches with the remote side, or sometimes I open channels to nodes with which I don't have existing channels, and satoshis are also spent on that. So it seems like all earnings are being consumed.
But there's an interesting point. As you know, you can buy a channel on my website, and that server, LND, which manages it, has been showing me over the past few years that, in total, more than one Bitcoin has been accumulated from all the payments for opening channels. I keep track of these statistics; I don't have the data on hand right now, but it's roughly around one and a half Bitcoin earned already. This includes the amount I'm trying to cover for opening and future closing of channels, as well as the percentage I set in my source code based on the size of the channel.
So, in principle, you can probably earn money, perhaps by selling channel capacity rather than earning from commissions. Moreover, you have to consider that over the entire time my network has been running, which is about six and a half years, the difference between what I can get back when closing all channels and what I put into the system shows that I've lost approximately 3.3 Bitcoin. This includes several instances where my nodes were penalized and all funds were confiscated. There were also moments when there was a bug in the LND server, and breaches occurred. If memory serves, this amounts to about 0.8 Bitcoin in total. The rest consists of funds spent on commissions, opening, and subsequently closing cooperative or force-closed channels, which I was very actively opening in the beginning.
If I were to summarize, it seems to me that running nodes solely to earn from routing doesn't make much sense. I believe the future lies with nodes operated by merchants aiming to accept bitcoins, as well as nodes set up specifically to pay for various services. Both types of nodes will strive to maintain channels with other counterparties, making routing more of a side activity rather than a primary focus.
Fundamentally, I am an active user of the Bitcoin Lightning Network. I enjoy making payments wherever possible using this network. Therefore, my network of nodes essentially serves as hubs that I connect to with my wallets to pay for the goods or services I purchase.
Of course, I could shut down all these nodes, set up a small personal home server, open a few channels to the merchants I buy from, and that would likely work perfectly for me. However, on the other hand, I feel a significant responsibility for supporting the Lightning Network. As long as I'm able, I will maintain these nodes and support the network to the best of my ability because I really want as many users as possible to connect to this network and experience minimal issues while using it. Naturally, I always recommend my friends and acquaintances to connect and open channels with my nodes. I often don't tell them that these are my nodes, but I always recommend them because I know that if a wallet is connected to any of my network's nodes, there won't be any problems when making payments for goods or services, and transactions will occur with the lowest possible fees.
However, I'm concerned that my entire network relies on me as a single individual. I realize that if I were to fall ill or something were to happen to me, there is a significant risk that the funds in the channels could become stuck, and my family and relatives wouldn't be able to retrieve them from the channels. Moreover, at some point, the hosting service might simply shut down the server due to non-payment, leading to an unpleasant situation. Additionally, I don't want to trust the management of the nodes to hired employees because where there is money and assets, there will always be people looking for ways to take them for their own benefit.
Thanks for sharing and discussing this so openly. I totally understand your struggle. The lightning network would definitely be a different one without you and your nodes. Thanks ❤️
Yes, gossip is a terrible way to propagate real-time information about the state of your channel. This is kind of by design since we stagger gossip, but also there are real propagation issues which we'd like to solve a gossip v2, but it will still be very slow. There's another proposal which is to allow you to publish a fee rate card that would have several different levels and the fee would vary depending on the remaining capacity in your channel. So someone just looking for a cheap payment would try the lowest fee rate that you offer and probably fail, but somebody who was really eager to get their payment through would be able to pay the highest.
at a certain moment I was thinking the same (if I understand correctly what you said): having a channel full, being then able to forward a large payment, is something desirable and of value, so the fee should be high.
but this is counterintuitive (and possibly wrong!)