Standards in Predictive Analytics: R, Hadoop and PMML (a white paper by James Taylor)

James Taylor (@jamet123) is remarkable in capturing the nuances and mood of the data analytics and decision management industry and community. As a celebrated author and an avid writer, James has been writing more and more about the technologies that transform Big Data into real value and insights that can then drive smart business decisions. It is not a surprise then that James has just made available a white paper entitled "Standards in Predictive Analytics" focusing on PMML, the Predictive Model Markup Language, R, and Hadoop.

DOWNLOAD WHITE PAPER

Why R? 

Well, you can use R for pretty much anything in analytics these days. Besides allowing users to do data discovery, it also provides a myriad of packages for model building and predictive analytics.

Why Hadoop? 

I almost goest without saying. Hadoop is an amazing platform for processing predictive analytic models on top of Big Data.

Why PMML? 

PMML is really the glue between model building (say, R, SAS EM, IBM SPSS, KXEN, KNIME, Python scikit-learn, .... ) and the production system. With PMML, moving a model from the scientist's desktop to production (say, Hadoop, Cloud, in-database, ...) is straightforward. It boils down to this:

R -> PMML -> Hadoop

But, I should stop here and let you read James' wise words yourself. The white paper is available through the Zementis website. To download it, simply click below.

DOWNLOAD WHITE PAPER

And, if you would like to check James' latest writings, make sure to check his website: JTonEDM.com

Zementis/Datameer Webinar - Best Practices for Big Data Analytics with Machine Learning (View Recording)

Please watch the  Zementis and Datameer webinar entitled "Best Practices for Big Data Analytics with Machine Learning."

VIEW RECORDING

In this webinar, we demonstrate through an industry specific use case how to identify patterns and relationships to make sound predictions using smart data analytics. You will learn best practices on:

• Selecting the right machine learning approach for business and IT
• Visualizing machine learning on Hadoop
• Leveraging existing predictive algorithms on Hadoop

Big Data Scoring with UPPI for IBM Pure Data (for Analytics and Hadoop)

In-database scoring is one of the most straightforward ways to gain insights from Big Data. It is no surprise then that the Zementis Universal PMML Plug-in (UPPI) is now being offered for a variety of database platforms. These include IBM Pure Data for Analytics (Netezza), Pivotal/Greenplum, SAP Sybase IQ, Teradata and Teradata Aster. Zementis also offers UPPI for Hadoop/Hive, including IBM Pure Data for Hadoop as well as InfoSphere BigInsights. It is in this context that we travelled to Vegas to attend the IBM Information on Demand (IOD) Conference.

I must say, I am always impressed by the IBM universe of products and tools that are being offered for analytics (descriptive and predictive) as well as Big Data in general. Zementis had a booth inside the Pure Data exhibit area and next to all the Pure Data appliances. As you can imagine, traffic was solid not just because of all the blinking lights but also because the conference itself attracts a lot of people. I believe there were 14 thousand attendants this year.

Why in-database scoring? Well, simple. Not all analytic tasks are born the same. If one is confronted with massive volumes of data that need to be scored on a regular basis, in-database scoring sounds like the logical thing to do. In all likelihood, the data in this case is already stored in a database and, with in-database scoring, there is no data movement. Data and models reside together hence scores and predictions flow on an accelerated pace.

Why scoring in Hadoop? Big Data and Hadoop are somewhat synonymous terms these days, since the latter offers an important technological platform to tackle the challenge of analyzing large volumes of data. In fact, predictive analytics is paramount for companies to extract value and insight from such data. By offering the Universal PMML Plug-in (UPPI) for Hadoop, Zementis takes a big step in making its technology available for companies around the globe to easily deploy, execute, and integrate scalable standards-based predictive analytics on a massive parallel scale through the use of Hive, a data warehouse system for Hadoop.

UPPI brings together essential technologies, offering the best combination of open standards and scalability for the application of predictive analytics. It fully supports the Predictive Model Markup Language (PMML), the de facto standard for data mining applications, which enables the integration of predictive models from IBM/SPSS, SAS, R, and many more.

CIO Review: Zementis selected as one of the top 20 most promising big data companies

Selected by a distinguished panel comprising of CEOs, CIOs, VCs, industry analysts and the editorial board of CIO Review, Zementis has been named by CIO Review as one of the "Top 20 Most Promising Big Data Companies in 2013." Congratulations Zementis!

Read CIO Review - FULL ARTICLE

That comes as no surprise since Zementis is all about kicking down barriers for the fast deployment and execution of predictive solutions. By leveraging the PMML (Predictive Model Markup Language) standard, Zementis' products allow for predictive models built anywhere (IBM SPSS, KXEN, KNIME R, SAS, ...) to be deployed right-away on-site, in the cloud (Amazon, IBM, FICO), in-database (Pivotal/Greenplum, SAP Sybase IQ,  IBM PureData for Analytics/Netezza, Teradata and Teradata Aster) or in Hadoop (Hive or Datameer).

Predictive analytics has been used for many years to learn patterns from historical data to literally predict the future. Well known techniques include neural networks, decision trees, and regression models. Although these techniques have been applied to a myriad of problems, the advent of big data, cost-efficient processing power, and open standards have propelled predictive analytics to new heights.


Big data involves large amounts of structured and unstructured data that are captured from people (e.g., on-line transactions, tweets, ... ) as well as sensors (e.g., GPS signals in mobile devices). With big data, companies can now start to assemble a 360 degree view of their customers and processes. Luckily, powerful and cost-efficient computing platforms such as the cloud and Hadoop are here to address the processing requirements imposed by the combination of big data and predictive analytics.

Creating predictive solutions is just part of the equation. Once built, they need to be transitioned to the operational environment where they are actually put to use. In the agile world we live today, the Predictive Model Markup Language (PMML) delivers the necessary representational power for solutions to be quickly and easily exchanged between systems, allowing for predictions to move at the speed of business.  

Zementis' PMML-based products: ADAPA for real-time scoring and UPPI for big data scoring, are designed from the ground up to deliver the agility necessary for models to be easily deployed in a variety of platforms and to be put to work right-away. 

Zementis ADAPA and UPPI kick-down the barriers for big data adoption!

Predictive model deployment with PMML

Model deployment used to be a big task. Predictive models, once built, needed to be re-coded into production to be able to score new data. This process was prone to errors and could easily take up to six months. Re-coding of predictive models has no place in the big data era we live in. Since data is changing rapidly, model deployment needs to be instantaneous and error-free.

PMML, the Predictive Model Markup Language, is the standard to represent predictive models. Given that PMML can be produced by all the top commercial and open-source data mining tools (e.g., FICO Model Builder, SAS EM, IBM SPSS, R, KNIME, ...), a predictive model can be easily moved into the production environment once it is represented as a PMML file.

Zementis offers ADAPA for real-time scoring and UPPI for big data scoring which make the entire model deployment process a no-brainer. Given that ADAPA and UPPI are universal PMML consumers (accept any version of PMML produced by any PMML-compliant tool), they can make predictive models instantly available for execution inside the production environment.

Check out the Zementis website for details.

PMML, Big Data, and Hadoop: Predictive Analytics at Work!

Big Data and Hadoop are somewhat synonymous terms these days, since the latter offers an important technological platform to tackle the challenge of analyzing large volumes of data. By the same token, predictive analytics is paramount for companies to extract value and insight from big data. It is in this context that Zementis brings its standards-based predictive scoring engine into a variety of Big Data platforms, including the cloud as well as in-database. By offering the Universal PMML Plug-in (UPPI) for Hadoop, Zementis takes a big step in making its technology available for companies around the globe to easily deploy, execute, and integrate scalable standards-based predictive analytics on a massive parallel scale through the use of Hive, a data warehouse system for Hadoop, and Datameer, an end-to-end BI solution that works on top of Hadoop.

UPPI brings together essential technologies, offering the best combination of open standards and scalability for the application of predictive analytics. It fully supports the Predictive Model Markup Language (PMML), the de facto standard for data mining applications, which enables the integration of predictive models from IBM/SPSS, SAS, R, and many more.

UPPI for Hadoop/Hive

Hive makes it possible for large datasets stored in Hadoop compatible systems to be easily analyzed. Since it provides a mechanism to project structure onto the data, Hive allows for queries to be made using a SQL-like language called HiveQL.

Once deployed in UPPI, predictive models turn into UDFs (User-defined Functions). These can then be invoked directly in HiveQL. In this way, UPPI offers Hadoop users the best combination of open standards and scalability for the application of predictive analytics. UPPI for Hadoop/Hive delivers instant and scalable scoring for Big Data while retaining compatibility with most major data mining tools through the PMML Standard. It also brings brings the scalability of Hadoop to the execution of predictive analytics.

UPPI for Datameer

Zementis and Datameer have partnered to deliver standards-based execution of predictive analytics on a massive parallel scale. This joint solution combines the Zementis plug-in for execution of predictive models with the power and scale of Datameer, an end-to-end BI solution that includes data source integration, an analytics engine, visualization and dashboarding.

Datameer uses Apache Hadoop, a Java-based framework that supports the parallel storage and processing of large data sets in a distributed environment, as its back-end storage and processing engine to scale cost-effectively to 4000 servers and petabytes of data. It provides wizard-based data integration to integrate large datasets of structured and unstructured data, integrated analytics with familiar spreadsheet-like interface and over 200 built-in analytic functions and drag and drop reporting and dashboarding visualization for end-users. Open API's for data integration, analytics and dashboarding make it easy to access custom data sources, utilize advanced or custom analytics like predictive modeling as well as custom visualizations. Predictive Scoring for Hadoop - Advantages UPPI for Datameer delivers instant and scalable scoring for Big Data while retaining compatibility with most major data mining tools through the PMML Standard. Through its versatile deployment solution, the Zementis/Datameer partnership: Brings the scalability of Hadoop to the execution of predictive analytics Supports PMML to avoid time-consuming and expensive one-off predictive analytics projects Integrates data from multiple data sources and formats without complex data and schema mappings that are time consuming to set up and difficult to change Provides cost effective storage and processing of large volumes of highly granular data that predictive applications often require Brings together a 100% standards-based approach to analytics that lowers total cost of ownership and increases reuse control and flexibility for orchestrating critical day-to-day business decisions.

Model Deployment with PMML, the Predictive Model Markup Language

The idea behind this demo is to show you how easy it is to operationally deploy a predictive solution once it is represented in PMML, the Predictive Model Markup Language.

As a model building environment, I use KNIME to generate a neural network model for predicting customer churn. Once data pre-processing and model are represented in PMML, I go on to deploy it in the Amazon Cloud using the ADAPA Scoring Engine and on top of Hadoop using the Universal PMML Plug-in (UPPI) for Datameer. So, the very same model is readily available for execution in two very distinct Big Data platforms: cloud and Hadoop.



The easy of model deployment and interoperability between platforms is the power of PMML, the de facto standard for predictive analytics and data mining models.

Resources:

1. Download the KNIME workflow used to generate a sample neural network for predicting churn
2. Download the PMML file created during the demo

Zementis is proud to announce PMML 4.1 support

PMML 4.1, the latest version of the Predictive Model Markup Language, is loaded with new and powerful features. 

Zementis is proud to announce support for PMML 4.1 throughout its scoring products, including:

• ADAPA on Site
• ADAPA on the Cloud (Amazon and IBM SmartCloud)
• UPPI for in-database scoring (IBM Netezza, SAP Sybase IQ, EMC Greenplum) 
• UPPI for Hadoop (Datameer).

We have also updated our PMML conversion process so that it now converts PMML files from older versions to version 4.1. In this way, every time a PMML file is presented to ADAPA or UPPI, it is automatically converted to PMML 4.1.

  

Our support for PMML 4.1 includes:

1) Scorecards (including reason or adverse codes and point allocation for complex attributes)

2) Post-processing: you can now transform scores into business decisions as well as output generic data manipulation steps

3) Multiple Models: a powerful and yet simpler way for the expression of model segmentation, composition, chaining and ensemble, which includes Random Forest models

4) Is the model scorable? The "isScorable" flag was added as a way to flag models not destined for production deployment, but that are nonetheless an important part of the model building cycle

5) New built-in functions (for pre- and post-processing).

With this new release and version update, ADAPA and UPPI can be used not only for deployment and execution of predictive solutions, but also for data analysis and processing before model training.

  

If you have any questions about PMML 4.1 and all the features supported in our products, please make sure to contact us or feel free to check out our PMML 4.1 forum for detailed support information.

Agile Deployment of Predictive Analytics on Hadoop: Faster Insights through Open Standards

This joint Datameer/Zementis presentation given at the 2012 Hadoop Summit outlines the benefits of the PMML standard as key element of data science best practices and its application in the context of distributed processing. In a live demonstration, we showcase how Datameer and the Zementis Universal PMML Plug-in (UPPI) take advantage of a highly parallel Hadoop architecture to efficiently derive predictions from very large volumes of data.

Watch it now on YouTube: 

http://www.youtube.com/watch?v=r_g99-kP_BE

Session Abstract:

While Hadoop provides an excellent platform for data aggregation and general analytics, it also can provide the right platform for advanced predictive analytics against vast amounts of data, preferably with low latency and in real-time. This drives the business need for comprehensive solutions that combine the aspects of big data with an agile integration of data mining models. Facilitating this convergence is the Predictive Model Markup Language (PMML), a vendor-independent standard to represent and exchange data mining models that is supported by all major data mining vendors and open source tools (see figure below).

PMML is an XML-based language developed by the Data Mining Group (DMG) which provides a way for applications to define statistical and data mining models and to share models between PMML compliant applications. It provides applications a vendor-independent method of defining models so that proprietary issues and incompatibilities are no longer a barrier to the exchange of models between applications. PMML allows users to develop models within one vendor's application, and use another vendors' applications to visualize, analyze, evaluate or otherwise use the models. Previously, this was very difficult, but with PMML, the exchange of models between compliant applications is now straightforward.

Webcast: Predictive Analytics on Hadoop

UPDATE: Thanks for your interest in our joint webinar with Datameer: Predictive Analytics on Hadoop. If you were not able to attend or would like to watch it again at your own pace, just click HERE.


To extract value and insight from "Big Data", leading organizations increasingly leverage predictive analytics. By using statistical techniques that uncover important patterns present in historical data, companies are able to predict the future. In doing so, they become more precise, consistent and automated in everyday business decisions.

Please join the Datameer/Zementis webcast entitled Predictive Analytics on Hadoop: Gaining Faster Insights through Open Standards to learn to efficiently derive predictions from very large volumes of structured and unstructured data.

WHEN: Thursday, July 19, 2012, 10:00 am PT / 1:00 pm ET

Free registration 

In this webinar, we showcase the technical capabilities of the Universal PMML Plug-in for Datameer, a solution that combines open standards and Hadoop to reduce complexity and accelerate time-to-market for predictive analytics in any industry and for any business application.

Leave this webinar knowing:

• The benefits of the Predictive Model Markup Language (PMML) standard as a data science best practice for data mining 
• How to leverage predictive analytics in the context of big data 
• How to reduce the cost and complexity of predictive analytics 

 You can register HERE

Agile Deployment of Predictive Analytics on Hadoop: Faster Insights through Open Standards

Join us for the 2012 Hadoop Summit at the San Jose Convention Center on June 13-14.

Ulrich Rueckert, Data Scientist at Datameer and Michael Zeller, Zementis CEO,  will be presenting on Wednesday, June 13, 1:30-2:10 pm.

Session Abstract:

While Hadoop provides an excellent platform for data aggregation and general analytics, it also can provide the right platform for advanced predictive analytics against vast amounts of data, preferably with low latency and in real-time. This drives the business need for comprehensive solutions that combine the aspects of big data with an agile integration of data mining models. Facilitating this convergence is the Predictive Model Markup Language (PMML), a vendor-independent standard to represent and exchange data mining models that is supported by all major data mining vendors and open source tools (see figure below).

PMML is an XML-based language developed by the Data Mining Group (DMG) which provides a way for applications to define statistical and data mining models and to share models between PMML compliant applications. It provides applications a vendor-independent method of defining models so that proprietary issues and incompatibilities are no longer a barrier to the exchange of models between applications. PMML allows users to develop models within one vendor's application, and use another vendors' applications to visualize, analyze, evaluate or otherwise use the models. Previously, this was very difficult, but with PMML, the exchange of models between compliant applications is now straightforward.

This joint Datameer/Zementis presentation will outline the benefits of the PMML standard as key element of data science best practices and its application in the context of distributed processing. In a live demonstration, we will showcase how Datameer and the Zementis Universal PMML Plug-in take advantage of a highly parallel Hadoop architecture to efficiently derive predictions from very large volumes of data.

Session atendees will learn:

• How to leverage predictive analytics in the context of big data
• Introduction to the Predictive Model Markup Language (PMML) open standard for data mining
• How to reduce cost and complexity of predictive analytics