Blog

Restcomm Scalability and High Availability

Restcomm is a next generation Cloud Communications Platform to rapidly build voice and text messaging applications, using mainstream web development skills. Restcomm lets companies use the Restcomm Visual Designer or any popular programming language to build voice, video and messaging applications. Restcomm does the heavy lifting by integrating with obscure Carrier Networks behind the scenes. Restcomm exposes clean and powerful APIs for web and mobile phones, so companies can run their telephony applications alongside other IT apps.

A frequently asked question from our customers is how Restcomm can scale horizontally on a Cloud Platform. Telestax provides the toolset and expertise to make this complex problem as simple as possible to solve.

Let’s cover at a high level how we approach High Availability and Scalability:

Multiple TelScale Restcomm Servers can run in a cluster across one or more geographically distributed data centers to scale up throughput and provide various levels of redundancy and high availability. The typical Restcomm cluster topography is as following with all traffic between the nodes being SIP and HTTP.

Restcomm Cluster Topology

TeleStax Converged Load Balancer can be used to achieve 99.999% reliability. The TeleStax Converged Load Balancer has the capacity to gracefully shutdown any node of the cluster for maintenance operations and to add or remove nodes from the cluster as load grows or reduces without any downtime. TeleStax Converged Load Balancer also allows SIP and HTTP Sessions affinity to a given Restcomm node in the cluster. This affinity can be controlled through a pluggable algorithm mechanism for the highest flexibility.

All Restcomm instances share the same SQL Database configured to serve distributed read and write queries. Call Detail Records and other Big Data structures are stored outside the shared SQL database. 

High performance is often a reason to implement a horizontally scalable Restcomm deployment. Its not always simple to predict the performance of an application prior to its actual testing in a realistic environment. However a few base metrics are useful for initial planning.

Our tests show that a single instance of Restcomm on a machine with 8GB Ram on a i5 CPU can handle the following traffic:

1. For conferencing application: 480 Concurrent calls with a rate of 90 calls per second and average call duration of 2:30
OR
2. Short IVR application: 200 concurrent calls with a rate of 80 calls per second and average call duration of 40 secs
OR
3. VoiceMail application: 100 concurrent calls with a rate of 50 calls per second and avarage call duration 1:30 minutes

The following video walks through an actual HA deployment of Restcomm with dynamic, real-time auto-scaling in action. This setup was demonstrated live at MWC 2015 on a real world mobile operator network (TruPhone).

 

Autoscaling of an Open Source Telco Cloud Network at MWC!

11025355_10153079036249654_1872581958_o (1)

As always Mobile World Congress 2015 was an hectic event, not only for TeleStax but from most of the partners as well. As described in this post, using Zabbix’s open-source enterprise monitoring software, TeleStax RestComm and MetaSwitch Project Clearwater IMS Core nodes were autoscaling thanks to Canonical’s Juju orchestration tool and its Ubuntu operating system as the system was being put under stress using SIPp load testing tool as shown in the video below.


Want to Learn more about auto-scaling Restcomm ?

Submit this form to request more information about autoscaling TeleStax products.




More partners and customers demos !

Booth visitors were also able to place a call to a UK-based phone number on Truphone’s network. The call was routed through Metaswitch’s Project Clearwater open-source IMS platform into the interactive voice response (IVR) system running on TeleStax’s RestComm all running on Amazon EC2. Fairwaves provided handsets with Truphone SIM cards, in addition to its UmSITE ruggedized base station, to allow users to connect to a local radio network. A Full Open Source Telco Cloud Network !

While preparing those demos, our partner Applicata’s very own JNTestsuite product caught a bug in RestComm days before the event and then was running continuously during the event to validate the system was up and running.

In addition to partners, it was also great to see some of our customers showing great demos, on their booth, of their products powered by TeleStax software such as a live WebRTC service interconnected to the PSTN

11028421_10153075042909654_2024563061_o

or a full Business Services Suite

11029989_10153078793079654_1247657253_o

TeleStax as WebRTC Meetup Panelist

Last but not the least, TeleStax was proud to be invited by Quobis and Dean Bubley to talk at the panel “WebRTC and open-source” on the great WebRTC Meetup (MWC Edition) at Poblenou campus Auditorium – Universitat Pompeu Fabra, which about 200 people attended !
The panel was actually loaded with great companies (Kamailio, Temasys Communications, Voxbone, Zingaya, Acision, Matrix.org) and the discussion heated up among panelists when talks about standardization of WebRTC signalling started which made it really unique and far from the boring usual agreements.

MWC feels like a marathon but this is the once a year chance to meet the community, partners, customers, prospects and friends that’s too good to pass on, looking forward to see you all again next year !

The myth about Telco API Security

One of the key concerns for opening telco APIs for calls and messaging to developers is malicious and fraudulent usage. Telecoms need to invite developers to build new revenue streams from value added services, but their core networks are not build for that. Traditional telecom equipment is inherently insecure. It was designed to be hosted behind vault doors. Telco API Security is a legitimate concern.

Even for internal IT organization of a telecom it has been incredibly difficult to have any kind of access to the core network. The core network has been treated as the golden goose that no one but the most privileged royalties can possibly see and even fewer blessed ones can touch. 

But its a different day and its time to fight or flight. So how do telecoms open up to new revenues without compromising their main infrastructure assets accumulated over many years of investment?

First Generation Telco API Security

There are several popular levels of security:

Most modern Telecom API products offer adequate security support from the list of technologies above. This is a giant step in the right direction as compared to older generation network equipment for SS7 and SIP. Most causal phone users still do not realize that almost all of their phone calls are easy to intercept and record, because they run over old, inadequately secured infrastructure – both landline and wireless.

Hopefully Telco APIs and internet telephony standards such as WebRTC will continue to grow in popularity at a fast pace. That will shift more of the communication traffic from insecure incumbent equipment to modern properly protected media. Realistically the shift will last for many years, probably decades.

In a hybrid environment of coexisting, interconnected communication networks, it is understandable that telecoms continue to be paranoid about opening up access to mainstream developers. Malicious developers can write offensive apps that can potentially bring parts of a telephony grid down. 

Restcomm App Store Quality Funnel

We associate the security layer above with First Generation Telco API providers. They are essential and highly recommended. They are far more secure and fraud proof than any traditional security layers in plain SIP and SS7 networks. Yet, more can be done to foster a healthy, safe and thriving communication app ecosystem.

Telestax provides a new layer of quality, security and trust via the Restcomm App Store. It implements a Quality Funnel, which is a proactive collaborative filtering approach that is proven and widely used by many eCommerce stores. Leveraging the combined power of participating networks, its efficacy increases as the square of its total user count. Much faster and cost effective than any individual network can achieve by itself.

The components of the Restcomm App Store Quality Funnel are as follows:

  1. Open Source Sandbox. Experimentation and testing of new applications is provided via Open Source development and simulation tools. While providing full featured software stack to developers, this approach protects operators from incurring unnecessary risks and expenses associated with running and managing a developer API portal. The costs are not only rooted in software licensing and operations, but also in the personnel expenses related to helping developers via online forums.
  2. Vendor registration review and approval. Each developer who is ready to offer a commercial app on the Restcomm App Store goes through a vendor registration and approval process. Telestax screens and interviews new vendors to ensure authenticity of their offerings. This process can take from several days to several weeks. Once approved, the vendor can publish apps to Restcomm App Store. Because of the trusted relationship between Telestax and partner operators, vendors do not have to undergo additional extensive approval process typically required by telcos for each of their vendors. Sometimes country specific procedural steps make take place as needed, but they would be in addition to an already established vendor profile, not completely starting from scratch.
  3. Application review, testing and approval. Once a vendor is approved, they can submit new applications to RAS (Restcomm App Store). Each new application and each new app version is reviewed, tested and approved by a Telestax quality assurance team.  Once approved, an application is published in RAS and available for download in all participating partner telco networks.
  4. User community ratings. Via open user ratings system, RAS apps are kept up to an honest standard that users have come to expect in modern eCommerce sites.
  5. Flagging offensive apps. End users as well as automated fraud detection systems can flag an app as inappropriate or offensive. After an app is flagged several times it is temporarily disabled for download and the vendor is alerted to address the issue in a timely manner. 
  6. Delisting app from App Store. If an app continues to be flagged after a reasonable time limit given for correction, all users running the app are notified and given the choice to disable the app. In certain extreme cases of detected fraud, the app is automatically disabled globally across all running instances on all partner telco networks. Finally the app can be delisted from RAS until the vendor provides adequate correction.
  7. Vendor reputation scoring. RAS maintains a dynamic reputation score for each. It is constantly re-evaluated and updated based on a variety of factors such as app ratings, frequency of app improvements, user reviews, generated revenue and others.

Telco API Security - Restcomm App Store Quality Funnel

Here Come the Developers to your Telecom

Developers Are Building Telecom Apps

 

I was talking with a new company the other day who are building a service that adds a unique service to existing telephone numbers.  The idea itself was interesting, but more interesting was who was on the team.  The founders were not telecom guys.  They were successful app developers in the financial sector with no previous communication app experience.

I’m seeing this more and more.  Developers are starting to understand that there’s a huge opportunity to make communication better in most every field.

No SIP, No Problem

It used to be that you had to understand telecom protocols like SIP in order to build communication applications.  Furthermore, managing the infrastructure was both too complex and too expensive for the average application developer.  Telecom was this old, crusty infrastructure that no one wanted to touch.

Then, companies like Twilio, Tropo, Nexmo and Bandwidth came along and de-mystified the old telecom protocols into web-friendly APIs that any developer could use.  Voice and messaging could now be a part of any application at a relatively low price.  WebRTC took us a step further, making it simple to integrate voice, video and data into any application.

However, as we’ve seen in other industries, while having the toolkits available makes the functionality more accessible, it is the ability for a developer to build an application and gain access to a wider marketplace that pours gas on the ecosystem fire.  Whether it’s Salesforce.com’s App Exchange, Apple’s App Store or Google’s Play Store, the pattern is now well-established:  Provide developers with not only the APIs and functionality but also access to the customer channel and they will maximize the potential of the platform.

RestComm‘s new App Store bridges the gap between carriers and developers by enabling carriers to quickly deploy a developer-friendly app platform within their network.

Why Carriers Win

  1. New revenue sources – By enabling RestComm App Store on their network, carriers tap into new revenue streams that were not previously available.
  2. Shared learning – Carriers can easily see what apps are doing well on other carrier networks and provide them for their customers.  By understanding what customers are willing to pay for, carriers can make intelligent decisions about where to invest more heavily in the future.
  3. Focus – By not having to develop their own application platforms from scratch, carriers can continue to differentiate and invest in underlying services while not foregoing new opportunities.
  4. Security – RestComm screens all applications prior to distributing them to the App Store network.  Carriers get the assurance that the apps running on their network are safe, and the carrier’s customer information remains private to that carrier.  If an app is proven to be unsafe, it can be instantly deactivated from all participating carriers.

Why Developers Win

  1. Write once, sell everywhere – Developers get access to any carrier running a RestComm App Store instance.  This means access to a global customer network right out of the gate instead of having to build applications for each carrier and sell them individually.
  2. Attractive revenue sharing model – RestComm’s typical revenue split is 70/20/10.  Seventy percent goes to the developer, 20% goes to the carrier, and 10% to RestComm.
  3. Low cost of distribution – Because developers don’t have to (necessarily) host any infrastructure for their application, the cost to get to market can be remarkably lower, as well as the ongoing support costs.  The carrier manages the infrastructure and the developer manages the application.
  4. Access to carrier sales channels – Developers leverage the carrier’s sales and marketing efforts to get more users on their applications, rather than bearing these costs by themselves.

Want to know more about how RestComm App Store can benefit you?  Get in touch using the form below and let us help you get developer friendly!


Talk to Scott about the Future of Carriers

Submit this form to request a private discussion with Scott Barstow on this strategic topic.




Are You Developer Friendly?

Slack, Hall and WordPress P2 are killing email.  Texting and messaging are killing the traditional phone call.  Video is becoming ubiquitous and expected.  Providing context for all communication will be the norm going forward. Metered services will quickly fade to black as the marginal cost to deliver service approaches zero.

By enabling the RestComm App Store, carriers can continue to innovate on core communication services and let application developers figure out how best to combine those services into meaningful end-user applications.

 

Applicata JN Test Suite Platform Catches a Bug in Restcomm Days before MWC 2015

Due to the humongous interest in the Dangerous Demo (video recording) originally shown at TADSummit 2014, TruPhone and its Open Source partners decided to work together on a new version for MWC 2015, which will be more spectacular and dangerous than ever. Due to the sophisticated nature of the demo, involving partners from cellular towers to core IMS network and app servers, the team decided to invite Applicata,  known for its cutting edge cloud telephony service testing product – Applicata JN Test Suite.

The Applicata JN Test Suite is an open platform for prototyping, automated testing and monitoring of services in GSM and IMS networks. In the context of the Dangerous Demo JNTS is utilized to run various testing scenarios against Restcomm applications. The main demo is happening at the Canonical booth in Hall 8.1 at App Planet Stand CC8.20. In-depth demonstration of the Applicata JN Test Suite can be found at MWC on Applicata’s Stand 7H10.

Demo booth visitors will be able to place a call to a UK-based phone number on Truphone’s network. The call will be routed through Metaswitch’s Project Clearwater open-source IMS platform into the interactive voice response (IVR) system running on TeleStax’s RestComm .

RestComm is a next generation Cloud Communications Platform. It allows web developers to rapidly build voice, video, WebRTC, USSD, SMS, fax and rich messaging applications. Restcomm opens new revenue streams for service providers by exposing existing SS7 and IMS core network assets to application developers.

The Bugs

In the process of developing the new demo, JNTS caught a bug in Restcomm which would have been difficult to locate by other means. In fact JNTS caught two bugs. One in Restcomm and the other one in the demo app that runs on Restcomm.

The Restcomm runtime bug was that when Restcomm was sending a SIP MESSAGE out to a registered SIP user, the message was populated with a local Restcomm host IP address, instead of an IP address that is reachable from the SIP client.
The second bug – the demo app bug would have been nearly impossible to catch prior to demo day, without JNTS.  The demo app used an absolute URL in one place for the demo. When the app scaled out with the underlying cloud infrastructure elements,  the Restcomm instance corresponding to the absolute URL went down, because of a zabbix misconfiguration. So the jnts tests which were successful prior to testing scalability started to fail, because RestComm couldn’t reach the absolute URL. Thus it was aborting the call way too early. JNTS caught that.

Before and After

For the earlier demo version used at TADSummit 2014, the team used sipp for automated load testing, but there was not tool available for functional testing of the integrated system. While sipp helped discover some performance bottlenecks, it did little to help simulate complete call paths the way that actual users would use the system. All non-trivial usage scenarios and edge cases had to be documented on paper and manually run each time any component of the demo system was updated. A very tedious and error prone process.

Sure enough there was a glitch during the TADSummit demo, which affected some of the users calling into the demo system. If we had JNTS available at the time, the glitch could have been prevented.

Amazingly the de-facto standard for end-to-end telephony system testing for most real world networks is no different than the labor intensive process used for the earlier demo version. Even the largest, well funded, multi-national telecoms write their test specifications on paper and expect solution providers to execute them manually as part of the acceptance criteria. Clearly a process with misaligned incentives.

JNTS changes that by applying Continuous Integration and Continuous Delivery best practices, long adopted by mainstream software developers,  to telephony networks and applications. The tool did not disappoint, as it caught two tricky bugs prior to MWC with enough time available to fix them. There is no 100% guarantee that everything will go smooth when the demo is live next week, but it sure feels better to know that a capable safeguard like Applicata JNTS is continuously watching out for problems.


Want to learn more about JNTS?

More details about the testing environment and the JN Test Suite script that implements the testing logic are described in more details below in Section JN Test Suite – Restcomm Demo Application Testing Environment .

Applicata JN Test Suite Overview

The current market dynamics raise extreme pressure upon network operators for continuous development and changes in technologies, introducing innovative services, establishing new partnerships, improving the quality of service and customers satisfaction.

With the increased competition between operators, customers become more sensitive about the quality of the provided services.

Mobile number portability service makes it easy for a complaining customer to move to another operator. Inbound roamers can switch to another local network even easier. This new competitive environment  raises operators’ interest in constant monitoring and control of their network and service quality. Proactively detecting eventual degradation as early as possible has direct impact on their revenues.

The key to overcoming the challenges is that the network and services should be regularly and pro-actively tested. A system capable of running automated tests, storing the results and analyzing the results is highly desirable.

JNTS provides a flexible and cost effective solution for automated functional testing of services in GSM and UMTS networks. It keeps these activities away from manual and time consuming testing, reduces costs, and produces measures and reports of service quality and availability

Manual testing is highly time consuming, costly and ineffective. Without special automated system testing activities are often neglected and this opens the door for bad network performance and bad quality of the provided services.

JNTS is easy to use with open functionality for service prototyping, automated testing and monitoring. It exports an interface for quick integration with third party management consoles.

Based on Telestax’ open source protocol stacks, JNTS is highly interoperable and adaptable to a wide range of network environments. The solution supports a wide range of protocols and can simulate the functionality for different nodes in the networks. This feature provides flexibility for running different tests initiated from the core network and/or from probes installed in the radio access network.

Typical Implementations

 Service Prototyping

JNTS is a quick, cost effective and easy service prototyping tool. The prototype can be effectively used as a proof of concept at early stages of a project. It can check and eventually modify the planned service application logic.

Network nodes simulation

New service development requires an environment where the developed service can establish controllable communication with the remote nodes in the GSM/UMTS/IMS networks.

XML scripts are created and deployed to JN Test Suite Platform for simulating the required behaviour of the remote nodes in the network that communicate with the service under development.

This scenario brings full featured and controllable network behaviour at the developer’s desk and eliminates the need to connect to a real network during the development.

Automated end-to-end and acceptance testing

A set of end-to-end and/or acceptance tests can be developed in XML scripts and deployed to JN Test Suite Platform.

The test can be easily run and repeated in automatic mode eliminating the effort, time and costs associated with manual testing.

Monitoring of services in production

XML scripts that simulate service requests can be created and deployed to JNTS, for constant periodic monitoring of services in production. The corresponding results will be collected and used for calculating service availability, KPIs, and SLA terms and conditions.

International Roaming Testing

Tests according to IR documents simulate roamers’ activity. Tests can run in probes distributed in the networks, or the activity can be simulated in the core network.

Applicata specializes in the design, development, installation and integration, of systems and software for telecommunications.

For more information on Applicata JNTS, please visit http://www.applicata.bg/jnts.html

JN Test Suite – Restcomm Demo Application Testing Environment

The following message sequence diagram describes the use of JN Test Suite for testing and monitoring the Restcomm Demo Application.

Sequence Diagram

The testing logic at JN Test Suite side is implemented as an XML script which is loaded and run by JNTS during testing.

The script with the test logic contains a finite state machine described using State Chart XML syntax. JN Test Suite platform expands the standard State Chart XML syntax with custom tags that cover telco protocol specific messages and parameters.

The State Chart XML script that controls the testing logic deployed in the JN Test Suite management console is shown in the next screenshot.

JNTS Script

The test result page in JN Test Suite management console is shown in the next screenshot.

JNTS Results

Telscale SMSC Gateway: Sending Messages from JSS7 to SMPP

smsc-logoSetting up a mobile network that processes SMS messages is not a straightforward exercise. This video tutorial will show you how you can setup a test environment that will enable you to send SMS messages from jSS7 to  SMPP through Telscale SMSC Gateway. You will also learn how to setup a SIP phone to receive and send SMS through Telscale SMSC Gateway

PART 1 – jSS7 to/from SMPP

 

 

Part 2 –  jSS7 and SMPP to/from SIP Phone

The second part of this video tutorial will show you how to setup a SIP phone and connect it to the Telscale SMSC Gateway.

 

Building USSD Mobile Application with Telscale Restcomm in Less than 10 Mins

ussd-gateway-logo USSD is a technology that has been around for a long time and there are millions of mobile devices that still support the technology. Building mobile applications that can span multiple generations of communication devices has never been easier. The video tutorial below will show you how intuitive it is to use Telscale Restcomm to build an interactive mobile application and integrate it with Telscale USSD Gateway.

 

The tutorial uses a Telscale jSS7 Simulator to send USSD PULL to the Application Server through the USSD Gateway. The whole setup takes less than 10 minutes.

 

Implementing value added services with Restcomm and SMS home routing

Business case for SMS Home Routing

SMS added value services is an exploding market. A number of successful services have been rolled out in various regions around the world. Examples include:

  • Ad supported SMS traffic
  • Location based marketing
  • Missed call alerts
  • Credit card transactions alerts
  • Loyalty programs for consumer brands
  • Personalized spam control
  • Political campaign updates
  • Multi-player word games
  • Sports news updates 
  • Delivering a message to multiple devices – desktop, laptop, mobile, tablet 
  • and many others

Different services are popular in different countries based on local demographics, culture and consumer habits.

Traditionally when a mobile subscriber of Telco A sends a message to another mobile subscriber who is customer of Telco B, the message travels directly from the SMSC server to the second subscriber. This leaves Telco B without opportunity to inspect the message before its delivered to their subscriber.

SMS home routing solves this problem. It is becoming increasingly popular around the world.

In a recent project with a Tier 1 international mobile operator customer based in Europe, we implemented SMS home routing network based on Restcomm and Telscale SMSC in a matter of days.

The new service will be soon rolled out to other regional operators in the group to enable added value SMS services for tens of millions of subscribers in 8 countries.

SMS Home Routing Technology background

SMS home routing also known as “Home Routing – Non Transparent Mode” was standardized in 2007 by 3GPP as specification 3GPP 23.840 5.2.3. However real world roll-out is still ongoing and its still considered a high-end feature for SMSC gateways. Fortunately for our customers its just a regular feature of Telscale SMSC. Readily available to all our customers to upgrade their networks in  a few days. 

For the technically inclined readers, the following diagram illustrates the SS7 network message flow introduced in the home routing spec:

SMS Home Routing SS7 message flow

SMS Home Routing SS7 message flow.

In a following article with a screencast we will share step by step details how we wired the MVNO infrastructure components to the MNO live mobile network. Then we will show how we quickly implemented several cool SMS apps with Restcomm.

Restcomm app using SMS Home Routing in action

Restcomm app using SMS Home Routing in action.

Design, Package and Submit an application to Restcomm App Store in less than 5 minutes

Have you ever though that it will take less than 5 minutes to design, package and publish a telephony application? That’s about as fast and simple as creating a web page.

This blog post will walk you through the whole process, just follow the video tutorials below:

Step 1: Designing an application

Step 2: Packaging an application

Step 3: Publishing an application to the store

Join live demo by Truphone of an all Open Source mobile network at #MWC15

Telestax invites you to see a live demo of open-source software written for the cloud at Mobile World Congress 2015 in Barcelona. The demo is happening at the Canonical booth in Hall 8.1 at App Planet Stand CC8.20.

The demonstration will show a new telco service deployment of a cloud-based IMS core and application server integrated with an open-source mobile base station and real mobile network.

Booth visitors will be able to place a call to a UK-based phone number on Truphone’s network. The call will be routed through Metaswitch’s Project Clearwater open-source IMS platform into the interactive voice response (IVR) system running on TeleStax’s RestComm . Fairwaves will provide handsets with Truphone SIM cards, in addition to its UmSITE ruggedized base station, to allow users to connect to a local radio network.

Using Zabbix’s open-source enterprise monitoring software, booth visitors will be able to see how a network operator can watch while Canonical’s Juju orchestration tool and its Ubuntu operating system are used to manage and scale the system, even as it is being put under stress.

The demo is one of a kind because it is showcasing a complete solution, working on a real service provider network, that is built entirely on open-source software, with the intent of showing how network operators can reduce infrastructure costs and get services to market more quickly.

Sign up for Telestax news, case studies and product updates:

Enter your personal information below: