Recently I saw that my Matomo reports were not showing the correct data. It seemed like the daily cron job wasn’t running or failing. To see what was causing this issue, I ran the archiving tool manually.
Error: Got invalid response from API request:
Response was 'PHP Fatal error: Allowed memory size of
805306368 bytes exhausted (tried to allocate 131072 bytes)
in /var/www/matomo/core/DataAccess/ArchiveWriter.php on
line 142 PHP Fatal error: Allowed memory size of
805306368 bytes exhausted (tried to allocate 32768 bytes) in
/var/www/piwik/core/Http.php on line 248 '
The archive script is reaching its memory limit of 805306368 bytes. Using more is not allowed, therefore the error. 805306368 bytes == 786432 Kbyte == 768 Mbyte. Somewhere a configuration is limiting the memory usage of PHP to 768 MB.
There are many, many configuration files for PHP available in my system. Matamo is using its own configuration, located at:
The file contains a parameter for setting a memory limit for archiving task.
minimum_memory_limit_when_archiving = 768
768 is exactly the value reported in the error. Increasing this value to 1024 (1GB) should solve the problem.
sudo vim /var/www/piwik/config/global.ini.php
minimum_memory_limit_when_archiving = 1024
OVA is a virtual appliance, ready to run on a hypervisor. With an OVA file, you can import the image into VirtualBox, VMWare, etc and all needed information is loaded from the file and you can start the VM. This works as long as your hypervisor is capable of reading an OVA file. Proxmox does not understand OVA, and you cannot use the image out of the box. Reading the provided VM definition is not possible. As an OVA file contains the VM disk, you can add the disk to a VM.
First, create a new virtual machine definition in Proxmox. You are going to import the disk image from the ova file, not the virtual machine definition. Therefore, you must first create a VM, this creates the necessary information in Proxmox, and then you are adding a disk to this VM.
The overall steps to add OVA image to Proxmox are :
Delete associated disk
Assign OVA to VM
Create a new VM definition
In Proxmox, add a new VM. Note the VM ID. You need this later when importing the OVA disk.
Go through the wizard to create a normal new VM.
It seems that you have to add a disk. The disk will be deleted later, the configuration entered here is not important.
I’ll use a CPU with 2 cores.
I am using the VM for SAP HXE, therefore I am going to use a little bit more RAM: 24 GB RAM in total.
After going through the wizard, the VM definition is ready and you can let Proxmox create the VM.
The new VM will appear in the list of available VMs in your server. Note the ID: 101 and the available storage locations.
Delete associated disk
Open the VM configuration and got to Hardware. The disk you added in the wizard is listed. This disk must be removed.
Remove the disk
Detach from VM
Select the disk and click on Detach. The disk state will change to unused.
Remove disk from VM
After the disk is detached, remove it from the VM. This will delete the disk file.
The next step is to import the OVA disk and assign it to the VM. As Proxmox uses LVM for managing its storage, a provided tool must be used to import the disk to LVM and assign it to the VM. Copy ova file to Proxmox server. Unzip OVA file. OVA is a zip file, you can simply unzip it to see its content. It contains the VM definition (ovf) and the vm disk (vmdk).
tar -xzvf hxexsa.ova
To import the image, you need to specify the VM and location where the disk is imported to. This information is available in Proxmox. You can see a list when looking at the server at the left menu. I am going to use local-lvm and VM HXE with id 101.
Some years ago I create a new instance in EC2 with the minimal configuration needed. The disk size of the root device and partition is set to 8 GB. Today I am reaching the limit of the disk size and need more space. Having the server in the cloud allows me to “simply” increase the size without having to buy a new HDD.
To increase the size of an EBS volume, you need to execute three tasks:
Resize file system
The commands to resize partition and file system are (gp2, ext4, t2):
You can use the EC2 console or CLI to extend a volume. I’ll use EC2 console. The volume used as root device for my EC2 instance is based on Elastic Block Store (EBS) and type gp2. This step is very easy to do, as you inform AWS that you need more storage and you get more storage assigned. You won’t be able to make use of that new storage as long as the file system isn’t resized.
Go to EBS > Volumes
A list of volumes is shown. Find the correct one using the volume ID. The root volume of my instance has 8GB size and type gp2.
To modify the volume, select the volume and then click on Actions > Modify Volume
The current configuration of the volume is shown. Last chance to verify you are changing the right volume.
I’ll only modify the size of the volume. From 8GB to 20 GB.
Confirm the change. Click on Yes.
In case AWS was able to assign more storage to your volume, a confirmation message is shown.
The size of the volume is now shown as 20 GB in the volume table.
I am going to do some work on my AWS EC2 instance that hosts my web site https://www.itsfullofstars.de. More precisely: I did the work already and it worked out well, that’s why you can read the blog Before starting the work, I wanted to have a backup of my data. The data is saved on a EBS volume and is also the root / boot volume / disk of my EC2 instance.
AWS has a nice documentation on how to create and manage snapshots. As always with this kind of generic documentation, it contains a lot of information, or too much, as all possible cases are covered. To have a simpler reference, I’ll show in this blog how I created a snapshot.
EC2: Instance with root volume on EBS. OS: Linux
Data: Size: 8 GB, type: gp2, SSD
Task: Create a snapshot of the root device
Note that it seems that you can create a snapshot of a root volume while the instance is running. AWS states that you should stop the instance first:
„To create a snapshot for an Amazon EBS volume that serves as a root device, you should stop the instance before taking the snapshot.“
Yes, 3 steps is all it takes to take a snapshot of a EBS volume used as root volume in a EC2 Linux instance.
Go to your EC2 instance and stop it. You can also log on to your instance and issue a stop command there. I am using the AWS console as here I can do everything without having to switch to another tool.
Select Stop, not Terminate, and confirm your action. Oh, yes, do not forget: afterwards your server is not online and its services not accessible. Plan for some downtime, communicate it, etc.
Instance state switches to stopping, meaning that the server is going to shut down. This can take a few seconds.
After the instance is stopped, the state is stopped. Now you can start creating a snapshot of your root volume, as it is not accessed anymore.
To create a snapshot, follow the stops outlined by AWS documentation. Go to create snapshot section in AWS console. In case you do not have any snapshots created yet, the list will be empty.
Let’s create a snapshot. To start, click on Create Snapshot. This will open a wizard. I wanted to create a snapshot of a volume, so I selected as type Volume and selected the volume from the dropdown list. It’s a good idea to provide a description.
To start the creation process, click on Create Snapshot.
The snapshot will be created immediately. Be aware: this means that the snapshot request was created, not the actual snapshot. Taking the snapshot / copy of the volume will take some time.
You can see the status of the snapshot creation in the column Status of the snapshot. It will be in state pending until all data was transferred from the root volume to the snapshot file.
Taking the snapshot can take a few minutes, depending on the size of your EBS volume. Mine was 8 GB and it took like 5-7 Minutes to create the snapshot. This was an initial snapshot, no delta. Only when the status changes to completed, the process ended successfully.
After the snapshot it taken, you can start the EC2 instance again.
During startup, the status of your EC2 instance will be pending. After completing, it is running and if everything worked without errors, your server and the services are back online.
In this blog I will detail how you use Microsoft’s AppCenter to build an iOS app und publish it directly to iTunes Connect. This allows you to decouple the building, testing and distribution process from the developers. The developer only has to push the app to the repository (I am using Azure DevOps) and AppCenter takes care of the rest.
You can add AppCenter features to you app, but it’s optional. I already have a running app that I just want to build and distribute. Next step is to configure the build.
Select the repository where the source code is hosted. I use Azure DevOps (free tier). Unfortunately, GitLab is not listed and in the free tier I am using it is not possible to add self-hosted git repositories.
AppCenter will connect to Azure DevOps via SSO and list the available projects.
This adds the repository to the build configuration. You’ll see the branches and last commit message.
To configure the build, click on the configuration option for the branch. The option will only appear when you hover with your mouse over the branch.
AppCenter will scan the project and find the available XCode settings.
You can configure the XCode version to be used for the build. This is very useful when you are using external libraries that do not work with newer XCode versions. For instance, the Fiori libraries included in my project were not released for 10.2.1 and the newer Swift version that comes with it. Therefore, the build exited with an error. Until SAP released an updated version of Fiori for iOS, I had to use XCode 10.2.
AppCenter offers options to automatically increase the build number, or run your XCTests.
To be able to send the app to iTunes, you must sign the build using your certificate and provisioning profile. I wrote two blogs on how to get these:
When you have these available, you can start configuring the app signing. You upload the files and provide needed credentials for your private key.
Next step is to define where you want to distribute the app to. You can send it to the official App Store, App Store Connect Users for your TestFlight beta testers, or to an internal Company Portal.
I am going to distribute the app to App Store Connect for TestFlight. Select App Store Connect. If you do not have yet an account linked to Apple, you can do this here.
AppCenter is connecting to App Store Connect and retrieves a list of apps. I only have one app available, making the selection easier. It also means that you have to create the app first in App Store Connect. AppCenter is not able to create the app definition for you.
I am using GitLab for private projects. GitLab is run using the Docker image provided by GitLab. I can access my instance from outside via a reverse proxy (Apache).
The setup is simple:
GitLab Docker container is running on NUC and listens on port 7080 for HTTP connections
NUC is connected via OpenVPN to the server on AWS
Apache as a reverse proxy listening on port 443 for HTTPS
Apache terminates SSL: incoming requests are HTTPS, but forwarded as HTTP to GitLab
Apache forwards incoming requests to GitLab on Docker
Standard setup of GitLab in Docker with Apache as reverse proxy will give access to GitLab without problems. Start GitLab container, configure Apache, done. You can access GitLab from the internet, create repositories, clone, push, etc.
While the setup will work out of the box, you need to carry out additional configuration for GitLab to really make it work with SSL terminating. What is not working correctly is:
The external URL is not configured, so the URL in the repository clone dialog is not using HTTPS.
You cannot upload attachments in the Wiki
You cannot add pictures in the Wiki via copy & paste from the clipboard
Uploading files / images may work in the issues dialog, but not in the wiki, as the wiki is using different upload service.
Attaching an image from clipboard fails.
My external URL is https://gitlab.itsfullofstars.de. Setting this value as external URL in gitlab.rb. You configure GitLab by setting the parameters in the file gitlab.rb and then reconfigure GitLab.
## GitLab URL
##! URL on which GitLab will be reachable.
Run reconfigure to enable the configuration.
This will set all parameters in all involved components of GitLab based on the values set in gitlab.rb. You can see the new value by looking at the automatically generated configuration file for the internal web server.
## GitLab settings
## Web server settings (note: host is the FQDN, do not include http://)
The problem is: GitLab thinks it is running standalone, with direct access to the internet. There is not a specific parameter to inform that the requests are coming from a reverse proxy with SSL termination. Setting the values in gitlab.rb will result in an erroneous configuration:
SSL for internal GitLab web server (nginx) is enabled
Nginx is not listening on port 80, only on 443
My Apache reverse proxy is configured to connect to nginx port 80. Hence the Service Unavailable error.
Port 80 is not working any longer. Accessing GitLab directly via 192.168.x.x:7443 on HTTPS port (Docker mapping 7443 to 443).
Access will work. GitLab tries to get a new TLS certificate during the reconfiguration process, but fails, therefore the self signed certificate.
Attaching an image won’t work
Because of the external_url value, GitLab will redirect to gitlab.itsfullofstars.de. As the reverse proxy is not able to connect, it’s a 503 error.
Configuring the external GitLab URLs results in:
An incorrect HTTPS configuration due to wrong certificate
Adjustment of Apache reverse proxy: no longer SSL termination
I do not want to take of managing GitLabs internal TLS certificate. I want to access it via HTTP only and use Apache for SSL termination.
The solution is to configure the external URL and to let the internal nginx run on port 80 and no HTTPS.
Check the configuration for the internal GitLab web server. The server should be gitlab.itsfullofstars, the port 80 and protocol HTTP.
## GitLab settings
## Web server settings (note: host is the FQDN, do not include http://)
Running reconfigure restarts the services, but if you want to be sure, restart GitLab.
My Apache configuration. Maybe not all parameters are needed, but it works.
RequestHeader unset Accept-Encoding
RequestHeader set Host "gitlab.itsfullofstars.de"
RequestHeader add X-Forwarded-Ssl on
RequestHeader set X-Forwarded-Proto "https"
Allow from all
After executing the above steps, your configuration should be:
I need the correct distribution certificate for the provided provisioning certificate. The provisioning profile contains a list of “linked” distribution certificates. If yours is not in the list, you cannot use your certificate to sign the app.
Which one is it? When I created the provisioning profile, I added a distribution certificate. I only have one, so this is the certificate I need.
To be able to use the distribution certificate in an external tool like Microsoft AppCenter, I have to convert the certificate into a p12 file. To convert, you can import the certificate into Mac Key Toolchain, export cert and private key and save as p12.
1. Download the certificate
2. Check file
The downloaded cer file is named ios_distribution.cer. The see if this certificate is for distribution just read the content using more. It must contain the line iPhone Distribution.
3. Import into keychain
Open the certificate in MAC Keychain.
I also have the private key for that certificate.
Select the certificate and private key and export both. Save the file and provide a strong key phrase.
Now I have my personal distribution certificate (Zertifikate.p12) and provisioning profile (.mobileprovision).
There are two ways to get the provisioning profile:
XCode automatically generates one
You create it manually
In case a single developer does everything from coding to uploading to the App Store from the MAC, it’s a good idea to let XCode handle the provisioning profile. For more complicated use cases like an external build pipeline, creating the profile manually is better or let the pipeline tool do everything for you (fastlane). Let’s take a look at each alternative.
In case of letting XCode handle automatically the provisioning profile, it can be found on your MAC. Go to folder:
I have there three provisioning profiles. To know which one to use, I deactivate and activate the automatic code signing in XCode.
Uncheck and check again the option. XCode recreates the provisioning profile and the correct one is the newly created file.
To create the provisioning profile manually, log in to Apple Developer Center. From the initial page to final profile it’s just 8 steps. You create a provisioning profile for an app and associate a distribution certificate to it. Only certificates assigned to the profile can be used to sign the app. Therefore, you need to create a new provisioning profile in case you add or change a distribution certificate.
Go to Certificates, IDs & Profiles.
Go to section Provisioning Profiles
Create a new profile.
Select App Store, as the profile will be used to publish the app to the Apple App Store / Connect.
Select the app.
Select the App Id you want this provisioning profile. This is the bundle id used in XCode (namespace). The profile will only be valid for apps using that App Id.
Select the developer certificates
The certificates added here can be used together with the profile. If your distribution certificate is not listed, you cannot sign and publish the app using the profile.
Give a unique name to the provisioning profile.
The profile is now generated and can be downloaded.
The new provisioning profile is listed in the Apple Developer Center.