General Troubleshooting Guide for Sixfab Connect Platform

If you encounter any issues while using the Sixfab Connect platform, the following steps may help resolve the problem:

1. Verify Browser Compatibility

Ensure you are using a supported browser (Chrome, Firefox, Edge, Safari) and that it is updated to the latest version. Clearing the cache and cookies or switching to an incognito window can often resolve minor issues.

2. Reproduce the Issue

Attempt to replicate the problem to identify the actions leading up to it. Note whether the issue is consistent or occurs intermittently.

3. Network and Connection Check

Verify your internet connection is stable. If the issue involves device connectivity, ensure the device is online and registered correctly in the platform.

4. Platform Updates

Check if the Sixfab Connect platform is undergoing maintenance or updates. This information can typically be found in announcements or status pages.

Troubleshooting ALPON Software Issues

This section provides guidelines to identify and resolve issues related to the operation of ALPON software. Follow these methods to diagnose problems effectively.

General Methods

Monitor K3S System Pods

Run the command:

kubectl get pods -n kube-system

A healthy system output should display all pods with the Status: Running and Ready: 1/1, as shown below:

If the STATUS is CrashLoopBackOff, Pending, or Unknown, this indicates an error.

If these pods are in an Unknown state, it could indicate that K3S is in a condition where it cannot recover its own system pods. One potential cause for this is a sudden power loss to the device. This situation can also cause the pods within the Sixfab namespace to enter an Unknown state.

Monitor Sixfab Namespace Pods

Run the command:

kubectl get pods -n sixfab

A healthy output should also show all pods with the Status: Running and Ready: 1/1. Example:

IIf the STATUS is CrashLoopBackOff, Pending, or Unknown, this indicates an error. If errors are found, proceed to inspect the specific pods:

Pod Details

Use the command:

kubectl -nsixfab describe pods <POD_NAME>

Replace <POD_NAME> with the NAME of the problematic pod from the previous command.

Pod Logs

Use the command:

kubectl -nsixfab logs -f -l app=<APP_NAME>

Replace <APP_NAME> with agent, modem-manager, network-manager, device-manager, or geolocation to view debug, info, or error logs.

Check ALPON System Services

1. Supervisor Service

Check the Supervisor, which is one of the system services of the ALPON software.

Use the command:

sudo systemctl status alpon_supervisor.service

A healthy status should show active (running). Example:

View logs with:

sudo journalctl -fu alpon_supervisor

In a healthy and idle state, the following repeating logs are expected.

2. Remote Terminal Service

If there is an issue with the Remote Terminal, check the Terminal service.

Use the command:

sudo systemctl status alpon_terminal.service

In a healthy output, the service should be active (running) as shown below. Any other status indicates a problem.

View the logs of the Remote Terminal service using the following command:

sudo journalctl -fu alpon_terminal

Common Issues and Solutions

Cellular Connection Fails

• Ensure the device is activated on the Sixfab Connect platform.
• The device's data limit may have been reached.
• Logs should be checked using the following command:

kubectl -n sixfab logs -f -l app=modem-manager

Logs containing result=false or Error may indicate a potential issue.

Device Not Active on the Platform

1. The device may not be able to connect to the internet. Ensure that Ethernet or Wi-Fi (if a registered network exists) is used to bring the device online. If this resolves the issue, check the modem-manager service logs as described under the 'Cellular Connection Fails' section.
   
   
2. If the device can connect to the internet but is still not active, it may not be receiving the required key and token information.
   
   
   • Check the Supervisor service logs using the following command:
     
     
     Shell

     sudo journalctl -fu alpon_supervisor
     

     
     
   • Also, check the contents of the definitions.yaml file. The values for the peer_details key must be assigned.
     
     You can access the file contents with this command:
     
     
     Shell

     cat /home/sixfab/apps/definitions.yaml
     

     Example content:
     
     
     Markdown

     peer_details:
       token: ****************
       wg_endpoint: ndev.sixfab.com:51820
       wg_peer_ip: ***********
       wg_peer_allowed_ip: **********
       wg_server_public_key: **********************
       websock_endpoint: wss://ws.sixfab.io/ws
       registry_url: cr.sixfab.com
     

     If the device cannot access these details, check with the technical support team to confirm if the device has been added to the Sixfab records.

Connection Lost After Device Update

• The status and logs of the Supervisor service should be checked.
• Performing the update process over a cellular network may lead to high data consumption. The data limit might have been exceeded.

Device Restarts Frequently

If the Device Manager application cannot feed the watchdog integration at regular intervals, the integration will restart the device. You can check whether the Device Manager is running with the command:

kubectl get pods -n sixfab | grep agent  

and view the application logs with:

kubectl -n sixfab logs -f -l device-manager

High RAM Usage

The RAM usage of the running applications on the device can be monitored with:

htop

Additional Notes

When a problem occurs, identifying the application or service causing it is crucial. Diagnosis can be based on understanding which software is responsible for each function:

Supervisor

• Manages and controls the overall Alpon software system.
• Updates systemd services and applications.
• Applies bundle updates initiated through the platform.
• Sets up the Sixfab Network by generating private and provisioning keys and retrieving Peer Detail information.
• Handles deployment operations such as adding, deleting, pausing, and resuming applications.
• Executes reboot requests sent from the platform.

Terminal
Provides the terminal feature for remote access through the platform.

Agent

• Establishes and maintains a stable WebSocket connection with the platform.
• Transmits data from the device to the platform via WebSocket.
• Sends periodic messages to inform the platform that the device is active.

Network Manager

• Manages the device’s network interfaces.
• Adjusts metric values for network interfaces, ensuring the modem interface (usb0) has lower priority than Wi-Fi (wlan) and Ethernet (eth) by default.
• Allows new Wi-Fi interfaces to be configured through the platform.
• Periodically sends interface data to the platform via the Agent.

Modem Manager

• Manages the modem by sending AT commands and interpreting responses.
• Ensures internet connectivity through the modem during the device’s initial boot, which is critical for operation.
• Handles LPA operations for adding or removing eSIMs via the platform.
• Controls the C LED on the device via the Device Manager.

Device Manager

• Sends periodic signals to the watchdog chip. If no signal is received, the device is assumed to be unresponsive, and a hardware restart is triggered.
• Manages peripheral components on the device, including LEDs, add-ons, ADC, buttons, I2C, and the TPM module.

Geolocation
Handles location services, sending the device’s location to the platform through the Agent.

By identifying which software or service is responsible, diagnosing and resolving issues can be faster and more precise.

Device Overheating

Possible Causes:

• Insufficient airflow around the device.
• Operation in a high-temperature environment.

Solutions:

• Place the device in a well-ventilated area and ensure that airflow around the device is not obstructed.
• Avoid operating the device in environments with temperatures exceeding recommended limits.

Network Connectivity Issues

Possible Causes:

• Incorrect network configuration.
• Weak or unstable network signal.
• Improperly connected antennas.

Solutions:

• Verify the network settings in the Sixfab Connect platform.
• Check the signal strength and, if necessary, reposition the antennas for better reception.
• Confirm that the GNSS, Wi-Fi, and LTE antennas are securely connected and not obstructed by any barriers.

Device Not Powering On

Possible Causes:

• The power adapter is not properly connected.
• Insufficient power supply.

Solutions:

• Make sure the power adapter is securely connected to the device.
• Verify that the power adapter is functioning correctly by testing it with another compatible device.
• We recommend using the power adapter included in the box.
• Ensure that the power supply meets the required specifications (minimum 27W). If using an alternative power source (USB Type-C PD, PoE), confirm it provides sufficient power.

Reporting an Issue or Requesting Support

If troubleshooting does not resolve the issue, follow these steps to report the problem to Support:

1. Gather Relevant Information

• Screenshots or Recordings: Take screenshots or record your screen to capture the issue, including error messages or unexpected behaviors.
• Browser and OS Details: Note the browser and operating system in use.
• Developer Console Logs: Open the browser’s developer console (Ctrl + Shift + I or Cmd + Option + I) to check for error messages and copy them.

2. Submit to Support

Contact the Support Team through the platform and include:

• A clear description of the issue and steps to reproduce it.
• Browser and OS details.
• Screenshots or logs.
• Any error messages from the console.

By providing this information, you help the Support Team resolve your issue more quickly.