OS4 and OS4 Pro high Packet Loss on DStar Reflector

WB7NIL

I'm the sysadmin for REF035. We have the reflector running at a Datacenter in Seattle WA, located next to switches for all the major internet providers.

Server specs:
Intel running CentOS 7 x64
60 GB NVMe
2 vCPUs
2 GB Memory
4 TB Bandwidth

When the reflector was moved to the datacenter, we started experiencing issues with those using the OS4s. They experience packet losses from 8% up to 90%. All other hotspots work great, including OS 1s thru 3s. There is something that the OS4s are sending that are causing excessive packet losses on the network. These are from OS4 from all over the country and not to any specific location. I've ran tests with the OS4 and the OS4 Pro on this reflector as well as on REF001, and on REF001, there is no issue.

When I use the SharkLink app on my OS4 pro on REF035, there is no packet losses. I had others try this as well so anyone with an OS4 Pro using SharkLink on REF035, can communicate without packet loss.

When I ran the reflector on a bare metal reflector, using a mikrotik router, I had the same problem, but was able to correct the issue by increasing the receive buffer on the router. In the case of the datacenter, I have no control over the network. This is a commercial network and I find it very interesting that the OS4 and pro are having issues on this reflector running at a datacenter.

So I need your help to try to correct this issue. What can I provide you so that may be able to troubleshoot what is going on with the OS4s. It seems they are doing something different than all the other hotspots out there. The issue is unique to the OS4s only and they should work on any network. It has nothing to do with the local routers they connect to as I tested this by going to another reflector.

I have exported logs from my OS4 Pro on conversations from REF035 and REF001, and they don't tell me much other than there are a lot of dropped/missed packets on REF035. REF001 looks good.

I also did a tcpdump on the gateway port on the server at the datacenter, but I see nothing that stands out. These files are available if you think they will help.

Please let me know. i really want to get this fixed. We have a lot of OS4 owners who are not very happy and have gone back to pi-stars and earlier versions of the Openspots until this is corrected. I have also heard that there are issues on REF055, but have no data to provide. There may be others having problems, but I've not heard anything concrete yet.

Anything you can offer or do to help to get this product to work on any network is appreciated. I'm here to help and if you need anything I can provide, please let me know.

Thank you.

Perry - WB7NIL
REF035/WA7FW/WA7DR DStar System Administrator
wb7nil@fwarc.org

NeilReinhardt

WB7NIL
Hi Perry,

Just found this about the Open spot 4pro. I'm in Australia and have just purchased this expensive device to find out that all my transmissions are dropping out of course no Internet connection problems. I'm putting it back to the Pro4. I'm very disappointed and don't know what to do.. I hope there is a fix or it's going back.
Neil VK4COM.

WB7NIL

Are you having problems on REF091C? I just heard that REF091C is having the same problem. No one else has reported any issues with the OS4 until now. I now suspect that there are more, but no one has reported it yet. The OS1s through 3's and the Pi-stars work fine on our reflector and most of our users have put the OS4s away for now, including me, where I have 2 of them. I had to get my pi-stars back out in order to use the reflector. Good luck!!

Perry - WB7NIL

NeilReinhardt

WB7NIL
Perry,
I have been using REF023C and some tests on other Reflecters as well, with the same results.
for the money, this is not good and Shark should fix this. Thanks for your reply.
Neil

WB7NIL

Neil, the interim fix is to use the SharkLink app, which works great, but that is not what we bought the OS4 pro to do. For me, it is an interim measure as I cannot send my back.

Perry - WB7NIL

NeilReinhardt

Do you think Shark would know about this problem? Neil

WB7NIL

So now we know REF035, REF091, REF055, and REF023, and as you state some others. I don't know if they will fix it, but I hope they will look at this and realize that there is a problem with how the OS4 works on DStar DPlus reflectors.

As you can read above, this started with me back in 11/2023, but I was able to fix it as I had control of my network. Now that my reflector is in the hands of a datacenter and I do not control any hardware or the network, the issue is that the OS4 does not work on all networks and this should be looked at. It is my hope that they will look at it and I am prepared to help them fix the problem. We have a growing OS4 DStar community (since the big Christmas sale) and that is when this problem really became an issue as more people picked up OS4s.

Let's see what the tech support has to say. I have pcap files and exported my logs from within the OS4 and will make them available to them if they want them.

Perry

NeilReinhardt

Let's hope it's fixed soon, Thanks for your time.
Neil

NeilReinhardt

Perry,
I just did a Beta upgrade and reports are a lot better now. I'm happy again.
Neil

WB7NIL

NeilReinhardt Better??? how much better? which version are you running now?

Can you come over to REF035C and give me a call so I can see what your packet loss looks like?

Perry

NeilReinhardt

WB7NIL
Beta V61 is available.

NeilReinhardt

Just called you on REF035c probably time difference. I will wait on 35c,,,, Neil

NeilReinhardt

NeilReinhardt
Perry,

I talked to a buddy on 30A for about an hour with 100% copy with no loss.
It has to be the servers. Something is wrong somewhere Else.
Neil

WB7NIL

NeilReinhardt Well if it is the servers, then why do all the other devices (pi-stars, Openspots 1-3 and all other Non OS4s) work ok? The servers are probably fine, it very well could be the networks that those servers are connected to that are having issues with the OS4s. If REF030 and 001 are at a datacenter where they are using a premium service @ a premium cost, then that may be why they have no issues. When I had my server running on a bare metal server, I had to make adjustments to my network (the router specifically) to fix the OS4 issue. Then I moved it to a datacenter and the problem returned. The software is the same, so it has to be the network, which in this case, I have no control over.

To have a single appliance that everyone has to adjust their equipment just to make it work on their network makes no sense to me. It should be plug and play and it should work on any network. The OS4s are definitely doing something different compared to the other hotspots where some networks just cannot handle what the OS4s are transmitting over the network.

REF030 and REF001 are big reflectors and I suspect that they probably designed and configured much differently and must more robust so that they can handles large amounts of traffic - much more than what I have or what you have in Australia.

So I do not believe it is the servers, but I do believe it is the networks they are connected to where the packet losses with the OS4s are high, but I think it is an issue with the OS4s that should be looked at and fixed so that they will work in any network environment.

Perry

WB7NIL

I see V61 is out. I will test this new version this morning and I am wondering what changes it provides. Maybe it addresses the issues with the OS4 and OS4 pro??

Perry

WB7NIL

WB7NIL Just tested.. no change. Not sure what the update fixes, but my issue is still there.

Perry

WB7NIL

Can anyone in tech support tell me what they need from me to help troubleshoot this issue? I have been working with my datacenter host, and they have made adjustments on the network and even migrated our server to a different host, and still, the OS4 and OS4 Pro has the same problem.

Please let me know what I can provide you to help get this issue rectified? It is not a single router or network issue. The problem is with folks who have OS4 and OS4 pros all over the country, and even in other countries connecting to our reflector. There is something that the OS4s do differently that our network cannot handle, and this network is a commercial datacenter.

Help please.

Thank you.

Perry

Tyrbiter

I think you need to provide the logs from the OS4, advanced mode means they will be output on the Status tab.

You can set up a terminal program and connect the OS4 to a PC USB port and get logging on that as well.

Finally you can use Wireshark to capture the OS4<->server packets by monitoring the WiFi network with a laptop, or your router's ethernet connection if you can enable port mirroring on a network switch if you have one.

WB7NIL

I've sent an email to SharkRF directly as I have not received any reply from Norbert on this forum.

WB7NIL

UPDATE: SharkRF Released V63 Beta which has vastly improved the quality of the connection of the OS4 and OS4 Pro on REF035. I contacted Norbert (Tech Support via email) and sent him logs from a conversation between myself and a user with an OS3 and OS4 Pro, showing the issues and differences between the devices.

Although the problem is not 100% corrected, reports are showing that users running V63 Beta have less than 10% loss and typically in the 1, 2 and 3% area of packet loss. The audio is intelligible with some blips and bips here and there, but otherwise usable. Whatever SharkRF has done in V63 Beta, they are heading in the right direction and the OS4's performance is almost that of the OS3 as far as packet loss on our reflector.

Thank you guys! It looks like you have just a few more tweaks of your code to make it 100%

Perry - WB7NIL