When running a CI job you may need to use some SAP tools. For instance, the MTA builder or Neo tools. Many CI servers include integration to build tools or plugins are provided by the community or vender. Jenkins offers plugins for Maven, Ant or Node that let you easily integrate these into a CI jobs. If you have a CI job for SAP, it is your task to make the necessary tools available. There are not many plugins for SAP available for Jenkins.
Some tools you may need can be found on SAP’s tool site. For instance, the MTA builder. A simple JAR file that is available for download and needed in case you are working with MTA apps.
Before you can download the JAR file, you need to agree to the EUL.
This means that you cannot download the JAR using cli:
Running the above wget command will not download the tool, but a web site. Some may know that this is very close to how Oracle protected it’s Java download. And the “solution” here is the same: send the right cookie via wget.
I have a git repository on SCP that I want to clone using git on my laptop. I thought this should be easy to do. The source code of my project is available in the git repo at SCP. Cloning the repo using git clone from this URL should work.
I did a), and b) did not apply, as I wasn’t asked for my SCN user ID nor password. SAP’s git troubleshooting guide contains a section about the error message. Good to know that there is a possible solution, but I already did already what the proposed solution to the error is:
„Ensure that you have the correct repository URL. Copy it from the Source Location section of the repository’s details page in the SAP Cloud Platform cockpit.“
As it is possible to access the repository in SAP Web IDE, it should also be possible to access it from outside SCP. I know that the git repository is protected. Maybe the requests from git cli is blocked by SCP? After all, I was not asked to authenticate. Maybe I can force SCP to ask me for my password? Changing the URL to include my SCN user ID did just that: I was asked to provide my password.
SCP is now asking for my password and – magic happening – the git service is now accessible and the repo can be cloned. Would be nice if the git service would ask me to authenticate instead of failing directly.
OpenVPN uses certificates to authenticate the server and clients. Therefore, the client needs to have a valid client certificate. This certificate needs to be issued by the CA server that also issued the certificate of the OpenVPN server. In my case, this server is installed together with the OpenVPN server on the AWS EC2 instance. The process to create the client certificate is the same as with the server certificate, only the certificate type must be client, or: TLS Web Client Authentication. This is done by specifying the client parameter in the generate certificate request command.
Depending whether or not easy-rsa or any other tool to generate a certificate request is available on the client, the request can be generated directly on the client. The vantage by creating the request on the client is that the private key will stay on the client. In my example, I’ll make use of the already available infrastructure on the OpenVPN server and generate the client request and certificate on the server and copy later the generated artifacts over to the client.
Create client certificate
Log in to the CA (OpenVPN) server and issue a client certificate request. The name of the client will be client1. Note that you can use a different name, like the FQDN of the client.
sudo ./easyrsa gen-req client1
As with the server certificate, give a passphrase and common name.
Next: sign the client1 certificate by the CA.
sudo ./easyrsa sign-req client client1
You need to confirm the signing request by entering yes and informing the pass phrase of the CA certificate.
The client certificate is now issued.
Private key: easy-rsa/pki/private/client1.key
Public certificate: easy-rsa/pki/issued/client1.crt
Move these files to the OpenVPN client.
OpenVPN client Installation
The client going to connect to the OpenVPN server running on AWS EC2 is a Raspberry Pi. The RP uses a Debian based Linux, therefore apt is used to install software. On the RP, install OpenVPN. Easy-rsa is not needed, as the CA is running on the EC2 instance.
sudo apt-get update
sudo apt-get install openvpn
Create a openvpn directory. Can be in /etc/ or in your user’s home. Put the client’s public certificate and privte key there. To use HMCA for additional security, copy the ta.key file from the server there too.
Copy the OpenVPN sample client configuration to your openvpn directory and edit the file client.conf.
cp /usr/share/doc/openvpn/examples/sample-config-files/client.conf .
Adjust the following lines to point to the correct server (AWS EC2) and local certificates and key. Example:
remote server.domain.com 1194
tls-auth /home/tobias/openvpn/ta.key 1
The tls-auth parameter is needed in case the server is configured to use HCMA. The shared key ta.key from the server is needed for this to work.
Start OpenVPN client
To start the OpenVPN as client, run the executable and pass the path to the configuration file as parameter.
You need to provide the pass phrase of the client1 private key.
The client will automatically connect to the OpenVPN server defined in the client.conf file (remote parameter) and the given port (1194). Make sure that on AWS EC2, this port is accessible for the client.
If all works, the client connects to the server and gets an internal IP assigned.
Recently I got some new hardware that I will use to run some useful software. To use the software from anywhere, I’ll need to have remote access. As I cannot do DMZ or port forwarding with my new internet provider, I decided to connect my home server using VPN to a access machine running on AWS.
The AWS EC2 Linux computer will serve as my entry point. Services running on the RP at home connected via VPN can be accessed from EC2. Other computers at my home cannot be accessed, as the IP is different and no route is configured.
This setup comes with several architectural questions to solve:
How to ensure the communication is secure?
How to guarantee the tunnel is up?
How to enable access from EC2 to the services running on the client?
The client must be assigned the same IP for the services be accessible from EC2
How to give access to the services from the internet?
The three top question will be answered in my next blogs about how to set up OpenVPN server and client. The first question is the easiest to answer: by using a VPN solution. I am going to use OpenVPN and this blog is about how to setup OpenVPN. I’ll cover the installation on the EC2 instance and on the Raspberry Pi, as well as the initial setup with the certificates, server and client configuration and how to connect. Starting the client and server as service keeps them running and in case the connection fails, an automatic reconnect is attempted. The EC2 instance can access the services running on the client automatically. The last two questions will be answered sometimes later.
Install OpenVPN on EC2
The OpenVPN software is available in yum on EC2 Linux AMI. You may need to enable the REPL repository. I assume you did this already. The packages to install a openvpn and easy-rsa.
sudo yum update
sudo yum install openvpn easy-rsa
This will also install a public key to install a package and ask for your permission to do so.
The easy-rsa package is needed to set up a certificate authority. In case you do have a CA available, you can use your CA to generate the certificates used by OpenVPN. For those that do not have a CA available, take the easy-rsa functionality.
The command above installs easy-rsa 3.x. With 3.x, the way how to use easy-rsa and to set up a CA and issue the certificates changed. You can see in detail how to use easy-rsa 3.x at the documentation available at the GitHub project site.
OpenVPN uses certificates, and easy-rsa issues those certificates. Basically, you have two components of easy-rsa to deal with:
Configuration of OpenVPN is put and read from /etc/openvpn. Easy-rsa software should be in a separate folder, like /home/ec2-user/easy-rsa, but to keep all in one place I’ll put easy-rsa inside the /etc/openvpn directory.
Note: for real productive usage, don’t do this. Separate easy-rsa executables and config files.
Copy easy-rsa to your selection location. For this, first find out where easy-rsa is installed.
repoquery -l easy-rsa
Location is /usr/share/easy-rsa/3.0.3. I’ll copy these files to /etc/openvpn/easy-rsa.
Follow the steps outlined at the easy-rsa git site. For the following steps, go into the directory where easy-rsa is installed.
sudo ./easyrsa init-pki
This will create the CA certificate to sign certificate requests. In other words: whoever gets access to the private key of the CA created in this step, can create new valid OpenVPN clients for your setup. Take care of the CA certificate and key.
sudo ./easyrsa build-ca
You’ll need to enter:
PEM pass phrase
The passphrase is used to unlock the private key and is an additional level of security. Even when someone gets a copy of the private key of your CA, without the pass phrase the key is not usable. The common name is used to identify the CA. I used the FQDN of my web server. After execution these two commands, the CA is initialized and can be used to issue certificates.
Generate Diffie-Hellman parameters.
sudo ./easyrsa gen-dh
Generate OpenVPN server certificate
The OpenVPN server needs a certificate issued by the CA to identify itself against the clients. This is a nice “feature” when using PKI. Server and client can validate the other side. Both need just to trust the CA certificate for this. The difference between the two certificates (client and server) is the included type. This is done by including an additional value in the certificate specifying the type of certificate:
TLS Web Server Authentication for the server and
TLS Web Client Authentication for the client
Which kind of certificate is going to be issued is specified by the easy-rsa command when creating the certificate request.
Generate certificate request
Create a certificate request containing the identity information of the server and let this request be signed by the CA. By specifying the server parameter, the request is for a server and the CA will include the value TLS Web Server Authentication in the extension.
sudo ./easyrsa gen-req server
As with the CA certificate, inform a pass phrase that adds additional security to the private key and a common name to uniquely identify the server. I used server as CN. Of course, it could also have been openvpn.mydomain.com or something else.
Send the request to the CA and sign it to issue a valid certificate. With that, the CA information is added to the CA, making it official and clients that connect to OpenVPN server will know if they can trust the server. Only when trust is verified, a connection will be established between the server and client.
sudo ./easyrsa sign-req server server
You’ll need to confirm the request by typing yes and the pass phrase.
The following certificate is needed to harden the overall security of OpenVPN. As OpenVPN is using TLS, it makes sense to add HMAC to validate integrity of the packages received. For this to work, a shared secret key is needed. This key will be written to a file named ta.key.
Take a sample configuration file as a template. Can be found in the doc folder of openvpn. The sample configuration file for the server is server.conf, and for the client, client.conf.
ls -1 /usr/share/doc/openvpn-2.4.4/sample/sample-config-files/
Copy server.conf to /etc/openvpn and edit the file.
sudo cp /usr/share/doc/openvpn-2.4.4/sample/sample-config-files/server.conf /etc/openvpn/
sudo vim /etc/openvpn/server.conf
Adjust the path to the ca, cert, key and dh files
These parameters inform OpenVPN where the certificates and Keys are stored. The CA cert ca.crt is used to validate the client certificates. They must be issued by this CA. The server.crt and server.key are used by the OpenVPN server to encrypt traffic and authenticate itselfs against clients. Diffie hellman dh.pem is used to provide Perfect Forward Secrecy.
Start OpenVPN server
To start the OpenVPN server and to test the current setup, run the following command:
sudo openvpn /etc/openvpn/server.conf
During startup, you need to provide the passphrase of the server certificate.
If all works, OpenVPN starts without erros: Initialization Sequence Completed. After this, the server is waiting for clients to connect.
If someone is reading my blogs for the last years you may remember that I have once written about setting up OpenVPN for accessing SUP on AWS. That blog was all about Windows and is outdated. I wrote it in 2012. But, as I published it once at SAP Community Network, it is not available anymore. SAP lost it during their last migration.
Rui Nogueira published a while back a blog series on SCN on how to implement an IoT scenario using a Raspberry Pi and HCP. I think the example shows very well how what the main use case of IoT is. When the blog was published, there was no SAP HCP IoT service available; if you want to implement the same example in a more correct way, you should use HCP IoT. Nevertheless, Rui`s example is easy to implement and shows how the different parts play together: client, server, user.
When I first came across Rui`s blog I noticed that he uses REST and goes through some effort to persist the data. I thought that it would be nice to adopt this to make use of OData. Took me some while to publish this blog J In the end, I did not adjusted his code, it merely served as an orientation. I wrote my own IoT server and client app. The result is a simple, clean and easy to read JEE app that uses JPA and Olingo for exposing the JPA entities and a Java client that does not need to be run on an IoT device. My user dashboard is very simple, implemented in D3.js, and only shows one sensor`s measurement data.
The client is a Java app that reads current weather data from openweathermap.org. To make this work, you`ll need an API key (free). In case you do not want this, I added a jMeter test that creates random temperature data (as seen in above picture). JMeter test file is located here: fish-with-odata\iotserver\test\jmeter\LoadData.jmx. The test is pre-configured to use localhost and port 7080. The test will run for 3 minutes as the 100 measurements are not created at once, but with a fixed time interval of 3 seconds.
iotserver, containing the server and user dashboard
Both are maven projects. It should not be a problem to transform them into Eclipse projects via mvn eclipse:eclipse, but while I developed both in Eclipse, I did not test transforming to an Eclipse project from maven. Sensor and Measurements are implemented using JPA. The relationship between both is that one sensor can have many measurement assigned, but a measurement can only be assigned to one sensor. In the Snesor class, this is done via @OneToMany
I am lazy so I let JPA decide when a measurement is created or updated. This may not be acceptable in most scenarios, especially when you depend on the exact time when the data was captured by the device and not when it was persisted in the DB. I implemented it that way to not have to take care of capturing the date in my client app and to keep the payload low.
To run the server:
mvn clean pre-integration-test
This will download the HCP SDK, install the server, run it on port 7080 and deploy the WAR file. After some while, the IoT server is ready.
$top parameter controls how many data points are returned. Beware that with OData, there is a page size defined that limits the max number of requests returned. This parameter is configurable in the class de.tobias.service.ODataSampleJPAServiceFactory
private static final int PAGE_SIZE = 50;
Assign any value to PAGE_SIZE you consider useful.
To run the client, you first must add your API key. This is done in the class de.itsfullofstars.iot. WeatherData. Add your API to APPID.
private static final String APPID = “YOUR API KEY”;
The SIT Brazil events offer a special service: videos. We do stream the videos during the event live and record them for on demand. For this to work I have set up architecture to support everyone involved: on site team, server team and the end user. The videos are recorded for later processing to publish them in high quality in our YouTube channel.
Our goal is to make the access to SAP related knowledge as easy as possible. In case you cannot join the event IRL, you can assist the session live and in high quality. In case you are at work and your corporate proxy blocks YouTube, you can assist the stream via the event app, or the event site. For later best is to use Safari, but VLC can also show you the stream. In case you cannot assist the event live, we`ll publish the session after a few days of processing them, adding the slides to the video, to YouTube. Who knows, maybe one day we will also offer the vídeos for download, together with the slides.
The software involved in the process is:
OBS: Capturing the video, save it locally and send it to my NGINX server
NGINX with RMTP: receiving the video from OBS and process it to the further channels: YouTube and HLS.
YouTube: YouTube live event. Streams the received video to the web.
HSL: Prepares the received video for HLS. This is done by using FFMPEG.
App: event app that connects to the HLS stream via HTTP.
Browser: connects to the HLS stream via the web version of the event site or to YouTube.
Several software components have to communicate with each other, on different protocols and ports, making it sometimes a challenge to set it up on site. To simplify to whole process, a central server hosted at AWS serves as receiver and distribution point to our channels. We only have to communicate with one server, and still can offer the stream in several formats. NGINX server is accessible under its own DNS name. Many companies do not like to give access to YouTube, and it is easier to get them open a port than YouTube.
AUDIO AND VIDEO
FMS URL: the server URL of NGINX.
Play Path: defines the name of the stream for internal handling at NGINX
File Path: location where the stream is saved locally for later processing, etc.
Easy. Once OBS is started, the input sources selected, we add some UI magic for branding and the stream is started. The target is the NGINX server in the cloud, the protocol and port is RMTP.
For the NGINX setup to work, I had to add the RMTP add-on. For this, I downloaded NGINX from git + the rmtp add-on and compiled the software. That`s easy to do:
Afterwards it is adjusting the configuration of NGINX, easily done using the attached file as a template: nginx.conf
What is the process flow in NGINX? The configuration file has a two room’s setup. One OBS can send its stream to one room only. To serve the stream of two rooms, two OBS setups are needed, while the same NGINX server can be used.
OBS1 -> /src/<key>
From there, NGINX pushes the stream to YouTube and to HLS. HLS is configured to make the stream available under /hls/roomN/<key>. The quality of the stream is only limited by the camera and upload bandwidth onsite.
“Subsonic is an open source, web-based media server. It is written in Java, so it can run on any operating system with Java support. Subsonic supports streaming to multiple clients simultaneously, and supports any streamable media.” (Source: Wikipedia)
My first contact with Subsonic was several years ago. If memory serves me right, it was around 2008 when I was looking for a media software that can be accessed from remote. At that time, Subsonic and the internet didn’t serve me well enough in Rio de Janeiro to continue my endeavor with Subsonic. Only in 2015 I came back to it, thanks to Raspberry Pi. This combination gave me a new look at media access. Up to now the experience I have is good enough to make me want to share it with others. If you want to stream your private music collection without spending money on a cloud based server / service, this blog may be for you.
Install Java 8
Subsonic wants Java 8, and Java 8 is available for Raspberry Pi. You can also download it form the Oracle Java website. The version you need is the one compatible with the Raspberry Pi processor: jdk-8-oracle-arm-vfp-hflt. Or you install it using aptitude.
Command: sudo apt-get install oracle-java8-jdk
This downloads the required packages
Afterwards, Java 8 is configured.
To test if Java 8 is available and correctly installed, just call Java.
Command: java –version
The output shows that Java 8 is installed. Congratulations!
Java is installed, but for applications to know where to find it, an environment variable is used: JAVA_HOME. This variable points to the install dir of Java. To not have to configure this for each user, the configuration can be made global to all. The above command installed Java 8 at this location: /usr/lib/jvm/jdk-8-oracle-arm-vfp-hflt
In case the file wasn’t saved as subsonic-5.2.1.deb, rename it. You do not have to, but it makes things easier.
The file downloaded above is a deb file. These files are meant to be used by the debian package manager and contain the actual file to be installed and dependencies.
Command: sudo dpkg -i subsonic-5.2.1.deb
This installs and already starts subsonic. To see the output log:
Command: sudo tail /var/subsonic/subsonic_sh.log
Not exactly what we want, as now sSubsonic is already running, but not configured. To stop subsonic:
Command: sudo /etc/init.d/subsonic stop
Subsonic stores its data in default folders. By default, for Debian it is /var/subsonic. Because subsonic was already started, this folder is created and filled with content, using the default subsonic user: root (yep, BAD, very BAD!).
Subsonic will be run in the background at start as a service. For this to work, a subsonic user needs to be configured.
Command: sudo adduser subsonic
Add the user to the audio group, in caes you want subsonic to output audio.
Command: sudo adduser subsonic audio
How to make subsonic use that user and run under that user id and not as root? The user information is stored in the default subsonic configuration file: /etc/default/subsonic.
Command: more /etc/default/subsonic
The last line must be changed to: SUBSONIC_USER=subsonic
Subsonic can now be accessed, but I want to be able to access it through my standard web site (this one). I want to do that without having to do much port forwarding or virtual hosts. The easiest solution is to make use of Apache as a reverse proxy.
As subsonic will be run from behind a reverse proxy, the standard URL will be different: the URL used will be /subsonic. Therefore, the configuration of subsonic must be made aware of that. To find out the correct parameter, take a look at which parameters Subsonic supports.
Command: subsonic –help
The parameter is context-path. This parameter must be added to the config file.
Configure Apache Reverse Proxy
Add the following RP rules to the config file of the virtual server:
For reverse proxy to work, the module must be enabled.
Command: a2enmod proxy_http
Command: sudo apache2ctl restart
That’s it from the Apache as reverse proxy part. Subsonic is already configured to use the new URL and Apache is ready.
To be able to use Subsonic from the internet, just start it and check that everything is working correctly. Start subsonic:
Command: sudo /etc/init.d/subsonic start
Command: ls -alh /run/subsonic.pid
Created as user subsonic
Command: ps -ef | grep subsonic
Log on to Subsonic.
It may be useful to transcode some music files on the fly. For instance, when the consumed bandwidth is too high, FLAC is used or when the user is accessing Subsonic over a low bandwidth network like 4G in Brazil. Subsonic allows for automatic transcoding of files. This feature can be activated for each user and the sampling limit can also be specified. It is therefore possible to define a user for mobile client usage and specify a max bitrate of 128 Kbps for him. The max bandwidth is defined in the user section of the configuration settings.
The programs ffmpeg and lame are installed automatically when Subsonic was installed via Debian package manager.
What does on demand portal offer and do you need it? Cannot the SAP Portal also be an on demand portal?
On demand is the “next big thing”: every product, every solution has to be available as an on premise and an on demand version. Simplified, on demand means that you can access your server and solution via the internet, from everywhere you are. For a normal user there is no difference in how to access a new on demand solution and how Yahoo Mail or Google Mail is accessed and used: enter the URL in the browser and start using it. For some solutions on demand is more a cultural shock than for others. Basically the main benefits for on demand are access, costs and maintenance.
SAP Portal users are familiar with web enabled access. Most of the time they are bound to the corporate network; sometimes they can access the services from outside the corporate network, by VPN or even by a “normal” URL. So where are the benefits of an on-demand portal http://wiki.sdn.sap.com/wiki/display/EP/SAP+Portal+On+Demand? Configure your infrastructure right and you can have an on-demand version.
The tricky part is the “your infrastructure”. Not every company does know how to do it right or even has the skills to do that in a secure way. The technology stack needed to run the SAP Portal is NetWeaver Java. There are stacks out there that are easier to maintain and that need fewer resources to run. You need a full J2EE stack for you application? Most portal applications only need a servlets container (like tomcat). The framework and standard UI of the SAP Portal are too heavy for Internet usage. Even with the External Facing Portal (EFP) framework, light weighted is defined differently. Licenses for the SAP Portal are cheap when your users are Business Suite users. As licenses are already covered, costs like bandwidth (if your company doesn’t have a flat rate or the money for enterprise grade backbone connection) and maintenance remain.
But still: problems that can be solved, so why an on demand portal?
Maintenance is where Basis surely will be relieved as the task for applying service packs and notes will be delegated and end-users will be happy too as a good on demand solution offers a higher availability than the infrastructure of a normal company can. Setup time and costs are inexistent compared to the on premise portal.
The ODP will be – naturally – an external facing portal (EFP). Considering the problems the on premise portal has when it comes to make it an EFP in regards to:
How will the ODP treat and solve these problems? And when you are an EP user, what kind of options will you get to use the ODP as your EFP? And will the ODP be the starting of the end of the EFP of the SAP Portal?
Looks like SAP is going to use the on demand portal to introduce a new stack to run the portal on. Open source based, OSGI support, something more like tomcat. The connectivity won’t be able to compete with what the SAP Portal offers, but as long as your backend exposes the data using HTTP/S it can be integrated; implying that you still have to be able to expose your backend data in a secure manner. If you know how to do that you can still opt for opening your corporate SAP Portal. But you won’t get the new SAP UI5. And that new interface alone justifies the on demand portal. Compared to the “old” SAP UI, UI5 was designed to be used over the internet in mind.
For the developer ODP is portlet development (WAR). It will be interesting to see if portlets developed for ODP also run on a native tomcat or on JBoss or on other competing products or what the effort is to make them compatible.
How will the access to information handled? A portal with portlets is just the visible interface to the user, but what about portal services? Will ODP come with a predefined architecture for accessing portal services and data?
What do I expect from ODP?
A new software stack, cleaner, easier, more open source and support of more and newer standards. The new SAP UI5. If everything works out well SAP will be forced to merge the two code lines of on-demand and on-premise portal. Refreshing thus to “real” SAP Portal too. What can be wrong about that? Mobile access is crucial. Of what help is a portal accessible from everywhere and you need a desktop browser? This should also drive the adoption of mobile access to SAP and the Portal on device http://wiki.sdn.sap.com/wiki/display/EP/SAP+NetWeaver+Portal+on+Device for the on premise SAP Portal.
As ODP gives us a revitalized portal running on new technology it should attract more developers. Done right developers have the freedom to choose how and with what they want to code: GWT, jRuby, PHP for Java, JSF, Java 5, 6 or 7, etc.
Open access to the information available at ODP. Everyone that already had to integrate the on premise portal – or the information stored and made accessible there – into another portal or product know that the SAP Portal is meant to be the last point of access. The SAP Portal’s primary design is to integrate content, but not to share it. Especially an ODP cannot be designed that way. As it is available 24/7 to everybody, so has to be the information.
So one problem remains: access. SAP has shown us more than once that this is a topic where SAP continues to deliver below the expectations. Currently, developing for and learning SAP on your own private environment comes with some constrains: downloading, installing, renewing the license every 90 days, and you cannot create your environment as you wish, you have to use what SAP gives you. (ex: CE 7.2). Not everybody can download several GB of data and install it; the hardware requirements are even today still a challenge for laptops – not everybody has more than 2 GB memory installed. Contrary to this, tomcat is downloaded and running in minutes. No wonder that tomcat is a popular servlets container.
It lies in the nature of on-demand that access isn’t a real problem anymore. The question is: will developers get free and no time limited access to ODP? To evaluate, learn and code the access does not need to be unlimited in all aspects: 1 or 2 users, limited bandwidth, CPU and memory usage, performance also does not count much, data base can be SAPDB. What counts is: give access to developers, from the very beginning.