Idevnews | Cloud Integration in 2011: Queplix Drives SaaS, On-Premise Integration with Data Virtualization

Idevnews | Cloud Integration in 2011: Queplix Drives SaaS, On-Premise Integration with Data Virtualization

Queplix Corp. is bringing its QueCloud data virtualization, integration and management technologies to the Salesforce.com AppExchange. QueCloud will let Salesforce.com users quickly and securely integrate with other leading SaaS and on-premise applications from SAP, Oracle and NetSuite.

by Vance McCarthy

Tags: cloud integration, SaaS, data virtualization, ETL, middleware, SOA, Salesforce, Queplix, QueCloud,

Queplix Corp. is bringing its QueCloud data virtualization, integration and management technologies to the Salesforce.com AppExchange. QueCloud will let Salesforce.com users quickly and securely integrate with other leading SaaS and on-premise applications from SAP, Oracle and NetSuite.

Queplix’s QueCloud uses data virtualization and metadata to simplify integration so users can simply select the data they want to integrate, share or align with Salesforce.

The company’s data virtualization and data management approach is designed to avoid costly and complicated ETL (extract, transform and load), and simplify data sharing and integration across multiple applications by eliminating programming, wire diagrams or even SQL.

“A lot of middleware and integration technologies remain glorified ETL tools to cope with a growing number of endpoints and connectors,” Queplix’s CTO, Steve Yaskin, told IDN. “On the whole, many [integration] solution providers are still largely just laying pipe or using ETL, stored procedures or working with SQL commands.”

After integration, Queplix’s QueCloud operations and automation will maintain data harmonization across the different applications, without the need for programming or manual data diagrams typically associated with traditional ETL technology.

Yaskin is a veteran of many point-to-point and ETL-based integration architectures, having worked for leading system integrators such as Accenture and Cap Gemini prior to joining Queplix as CTO.

Queplix created QueCloud with the premise that long-standing ETL technologies are about to hit a breaking point, driven by a skyrocketing growth of web-based and cloud-based SaaS and data sources, Yaskin told IDN.

“Companies are now, or soon will be, where the number of [integration] connectors is just becoming unmanageable. So, laying pipe with ETL just won’t scale to meet these newer architectures,” Yaskin said.

Queplix and QueCloud avoid laying pipe altogether by using data virtualization and metadata technologies. “Our simple idea is if we had more data about the data we need, we’d be able to make integration simpler because we would have accumulated more knowledge about how to get just the data we need to the right applications and end points,” Yaskin added.

“With our persistent metadata server as the central location we can reach out to multiple sources [and targets] without pipes.”

Steve Yaskin
CTO
Queplix Corp.

Queplix differs from ETL approaches in that it uses a persistent metadata server to provide end point and data integration in a hub and spoke architecture. “With our persistent metadata server as the central location [for data integration] we can reach out to multiple sources [and targets] without pipes,” Yaskin said.

Queplix does not move data nor create data marts – rather it “interrogates” enterprise data leaving it right where it resides. To do this, the company uses a series of intelligent application software blades that identify and extract key data and associated security information from many key enterprise apps. The blades contain software that “crawls” all data sources to identify the metadata and other information about the data.

“We extract metadata from the [data] and then build a metadata catalogue that describes all those objects into an object model, not a relational model,” Yaskin said.

From that, Queplix constructs a centralized virtual store (or what Queplix calls “a virtual catalogue”), in which all the data (and associated information) can be stored as objects. “This means a wide range of information can be presented as an object – data, tables, XML, SOAP, whatever – everything becomes an object in a virtual or metadata catalogue location.”

At runtime, the data virtualization-driven integration is accomplished via the Queplix Engine. Once integrated, Queplix also has offerings to keep data consistent between applications with its data harmonization automated data synchronization process.

So, for example, Queplix can connect to Oracle DBMS with 5,000-10,000 tables and what comes out is a set of business objects, represented by metadata, Yaskin explained. “We have an abstraction layer, which creates the virtual catalogue. This lets you integrate or transform just the data you want and put it where you want it – without the need to move any data over pipes,” he added. “So, when you want information about your customer, we have an object called ‘customer’ that you can share with other on-premise or cloud applications, or even use to make sure all fields in different applications are consistent.”

Queplix can work with relational, object and other structured data, as well as unstructured or semi-structured data, extract metadata and create XML structures and virtual mappings, he added. Within Queplix, this virtual mapping achieves integration by merging the metadata objects, which eliminates the need for manual data mapping tasks, especially for mapping fields, which can be required of many MDM projects.

“Queplix does not require data owners to make any changes to the way they handle data because our blades contain all the required metadata they need to interrogate their data sources and extract the metadata we need,” Yaskin said.

As a result of Queplix’s approach to data virtualization, metadata capture and automation, companies can reduce the lifecycle total cost of ownership for such data integration and management by up to 75%, compared with ETL tools, he added.

"Apps like these continue to push the social, open, mobile and trusted capabilities customers expect from the salesforce.com ecosystem,” said Ron Huddleston, Salesforce vice president for ISV Alliances, in a statement.

An analysis of the recent trends in the data management space points to the emergence of the SMB-driven master data management adaption. The ubiquitous MDM projects have always been a prerogative of the Global 5000 companies; however this has been changing throughout 2011. The tidal wave of data caused by the amount of the information heading to the SMBs and driven by increasing SMB participation in the social network ecosystem requires smaller companies to start looking at Data Quality, Data Governance and MDM within their organizations. With SaaS pricing going down and usability going up, SaaS applications like NetSuite, Salesforce and Jive are spreading through the market segment like wildfire. As SMB CIOs and CFOs start to gain the benefits of having a solid (but inexpensive) back office and sales force management systems, they inadvertently start experiencing the data management problems way ahead of their company traditional growth needs.

In a recent article for Information Management, The MDM Institute's chief research officer, Aaron Zones, outlines future MDM trends through 2013. 

For example, trend number two is “MDM Market Momentum,” but it includes all of the following trend predictions:

1.       While Global 5000 enterprises will spend an average of $1 million on MDM software, they'll spend $3-4 million more for integration services.

2.       IBM, Oracle, SAP and Informatica offer SMB's entry-level MDM for $250,000 to $500,000.

3.       Mergers and acquisitions, the drive for sales leads, and compliance will be the drivers for funding MDM.

4.       IT-initiated MDM projects will struggle to justify the business value.

5.       There will be a skill shortage for MDM and data governance projects, leading to more work for systems integrators through 2012.

We confirm these trends as we interact daily with Queplix prospects and customers. We sell our persistent metadata server to both SMB and larger companies and what we observe is that SMB’s and Global-1000’ requests for data management are essentially moving towards each other and starting to overlap. Similar features are being requested on both spectrums of the market driven by data management needs. While SMBs have a lot more interest in measuring their social outreach and networking than their larger counterparts, very similar trends and feature requirements emerge around data quality and data and application integration. Larger companies start to pay very close attention to their brand recognition and social “chatter” around their products and marketing activities, measuring people perceptions and even “friending” them directly. All these data needs to be merged and analyzed within the corporate structure. Naturally, larger companies have more data silos and larger data volumes to handle, but essentially this is where the differences stop.

Both SMBs and Global-1000s need to integrate their data from social networks, marketing campaigns, sales force and internal accounting systems to have an agile and holistic view of their data at any moment in time. Is it possible for them then to continue using 20 y.o. technologies like data warehousing or ETL to address these problems? The answer is a flat out no. The amount of data and ever-changing dynamic nature of data feeds and constant need to bring even more data for analysis renders traditional ETL and rigid mapping tools useless.

Big data management vendors i.e. IBM, Oracle and Informatica, are dropping prices to make MDM and data quality tools more affordable to the SMB segment, while trying to adapt the SaaS and cloud platforms to their traditional large enterprise offers. Their product management teams had to come to this realization by observing the market trends; they are seeing what we are seeing and their reaction is confirming our analysis of these trends. However, lowering pricing and adapting cloud platforms to products which were not designed for this is not the best approach to solve current data management problems. 

The MDM and data management industry overall is in desperate need of the next qualitative leap on its technology S-curve.We see customer environments, both on SMB side and larger companies, where several ETL tools are deployed to feed a data warehouse (or two), to create a datamart (or two) for BI; mostly just moving and shuffling data around and laying pipes. IT teams are stretched and consulting teams are brought in to close the gaps. Unfortunately, the requirements constantly change and IT departments armed with traditional tools are falling behind, causing out of budget run-aways, which never reach the destination. In the same paper above the author mentions more than a quarter of MDM expense is services. I think this is a conservative number because it is based on comparing initial estimates to the current point in time; however as MDM projects progress the amount of services required to implement traditional systems only increases. To learn more about how persistent metadata servers are addressing the SMB’s MDM market needs please go to http://www.queplix.com/solutions.html or download our software for free and give it a try. We will even configure it for you and teach you how to bake the bread, all at no charge.

Queplix Launches Cloud Service to Integrate Cloud Systems - Cloud Computing - News & Reviews - eWeek.com

Queplix Launches Cloud Service to Integrate Cloud Systems - Cloud Computing - News & Reviews - eWeek.com

Thoughts on Big Data and Data Virtualization

Big Data Analysis in Relationship to Queplix Data Virtualization Solution

On the plus side for obtaining IT and business alignment, more companies are beginning to combine business and information management responsibilities in a single role, carried out by a single person, rather than a “business and IT partnership” with two people, two hierarchies and two sets of reporting relationships. Gartner expects 20 percent of companies to employ business information managers by 2013, compared with 5 per cent in 2009.

-       Massive Data News in the report from April 2010

 

Here are the next ten things you should know about big data:

1.    Big data means the amount of data you’re working with today will look trivial within five years.

2.    Huge amounts of data will be kept longer and have way more value than today’s archived data.

3.    Business people will covet a new breed of alpha geeks. You will need new skills around data science, new types of programming, more math and statistics skills and data hackers…lots of data hackers.

4.    You are going to have to develop new techniques to access, secure, move, analyze, process, visualize and enhance data; in near real time.

5.    You will be minimizing data movement wherever possible by moving function to the data instead of data to function. You will be leveraging or inventing specialized capabilities to do certain types of processing- e.g. early recognition of images or content types – so you can do some processing close to the head.

6.    The cloud will become the compute and storage platform for big data which will be populated by mobile devices and social networks. 

7.    Metadata management will become increasingly important.

8.    You will have opportunities to separate data from applications and create new data products.

9.    You will need orders of magnitude cheaper infrastructure that emphasizes bandwidth, not iops and data movement and efficient metadata management.

10.  You will realize sooner or later that data and your ability to exploit it is going to change your business, social and personal life; permanently.

 David Vellante in Big Data on February 16, 2011

Queplix VDM solution provides a data management solution continuum, starting from data integration of multiple disperse data sources to Master Data Management. All in a single “dashboard” view. We have an automated NoSQL, object oriented representation of business objects, abstracted from multiple sources and described as metadata repository. QVDM is a persistent solution that operates with minimum disruption to the sources in automated fashion.

QVDM offers today enhanced Data Alignment, Data Quality and Data Enrichment, proactive Data Stewardship interface and Global Data Dictionary. As such, QVDM is a full-spectrum data management solution. One of our strengths is in our ability to intelligently identify and described business objects from a variety of data sources, using our Application Software Blades™. In doing so, we eliminate the need to deal with proprietary data storage formats and the need to copy large amounts of data in order to make it available for analytics.

The Big Data industry has been developing rapidly recently, even though the technology was created years back (Google Big Table, etc.) The goal of Big Data is to be able to store and processlarge amount of data for analytical purposes using Map/Reduce technology. The big data technology by itself does not provide analytical engine, but rather enables it to operate on large volumes of data. The big data storage vendors (i.e. Hadoop) provide flat non-relational storage facility and the multi-processing engine to address BI queries for large data volumes. It is not possible to achieve the same performance using traditional RDBMS.

The most obvious synergy between QVDM and Big Data technologies is in the noSQL approach to data management. Big Data vendors pursue common goal which is to enable BI solutions to work on large volumes. Let’s consider an example of Jaspersoft BI:

“Jaspersoft’s vision goes well beyond Big Data. Our modern architecture and agnostic data source support is tailored for the cloud, from IaaS to PaaS, either public or private variations. In particular, NoSQL support puts us in the driver’s seat to become the de facto embedded standard for reporting and analysis within PaaS cloud environments.”

— Brian Gentile, CEO of Jaspersoft.

Jaspersoft’s BI engine can be deployed on top of Hadoop and in order to work effectively with large data sets it needs to utilize abstraction of business entities. Queplix Virtual Data Manager is built on the noSQL architecture and can provide the abstraction required fro Jaspersoft and other BI vendors today.

QVDM architecture works natively with Big Data engine by abstracting the data from the siloed sources and can optionally be used to migrate the data to the Big Data storage like Hadoop. With QVDM it is now possible to create the abstracted metadata layer and then use it to recreate the business objects in Hadoop in order to copy/move the data and therefore enable BI to work on Big Data engines. In other words, through Queplix Hadoop application Blade, we can now enable BI vendors is to virtualize Big Data repositories and provide persistence. 

New NoSQL, Hadoop and Big Data Webinar!

http://campaign.r20.constantcontact.com/render?llr=ve7cot47&v=001QhuZciaLI3WEy4uhQOfPFH99H8ehg-2W5nMFMhLZuBu68BVt84cOSzI2K2Cb3gTsQauqh3wvJSQotBCwhVo50seC-SUKMVAa-xDqHXcYTJcqTtthraKsb809f6gzL3l8OWJYZVrQ2o4q3sReu7Cmq4Xq8R8T2cBIoSQ6dmIly0hNL2FJG-8q7N3qQHpGFsgGzDUi86nC1D6gup7-Yru91brAYl-LwyHjcDbNytmSVj1_wxXZgZToYrFxGHRkec1Pt9ZSDDFeJlGae8FXmrzQYzhOCAMF_5LEOrvRdEGz3tw%3D

The Chief Executive Officer of Queplix, Mark Cashman, shares his view and insights on the state of Queplix and trends within technology and industry that are shaping them.

The Chief Executive Officer of Queplix, Mark Cashman, shares his view and insights on the state of our business and trends within technology and industry that are shaping them.

A Market Point of View

Written by Mark Cashman Tuesday, February 01, 2011 11:11 PM

Some things should make you stand up and take notice; data virtualization is one of those things.  Certainly virtualization as a technology or category is no longer new.  But data virtualization is one of the emerging segments in virtualization that has now reached critical mass.  All of the same drivers make it compelling - lower cost, lower risk and higher return on investment.

Data virtualization is the process of identifying, packaging, moving and ultimately storing your data without physically making additional copies throughout the process.  Data virtualization creates a layer between applications and their data so that you can use the data, to your benefit.  This virtual layer allows you to unlock your data from proprietary applications to get the utility you need.  Data virtualization separates the applications from your data such that you can use the data in new ways.

Data virtualization has been cast by some as a replacement to ETL, EAI or EII.  It is far more substantial - data virtualization is enabling you to do things you have not been able to do before without a tremendous amount of work, and with greater return on investment and lower risk.

As a proof point I would suggest doing a little research on which companies are now marketing themselves as data virtualization vendors.  Here you  you will find some household names from the technology company roster. These vendors understand the challenges the enterprise face and want to cash in on the action.  But, of course, it is often the case of finding a wolf in sheep’s clothing.  Their primary business model is wrapped around legacy technology.  For some, the complexity of their approach brings so much customization that there is more revenue in the services than in the software product.  Surely the model is upside down - we can help you fix it.

So, do your research and consider taking on change.

Stand up and take notice of the challenges you have faced to date and ask yourself does the existing way of managing my data really work? Can I achieve my goals and objectives using existing tools and technology, and how long will it take? The comfortable or easy bet didn’t get you to where you are today. Reach out explore other opportunities and realize another evolution surrounds you. Don’t become extinct like the dinosaurs!  The nimble market leaders across all industries know how to use new and innovative technologies that offer competitive advantage.  Faster development cycles, lower risk and greater return on investment are more important than ever.  Data virtualization can bring you all of this across applications for data discovery, business intelligence, data integration, data management, governance, quality, compliance and much more

In my next blog post I will explore how this new approach streamlines many of the tasks you perform today simply and more cost effectively.

Queplix Data Virtualization and Hadoop - marriage made in heaven

Recently, there have been a lot of news and development covering advances in parallel processing frameworks, such as Hadoop. Some innovative data warehouse software vendors are increasingly starting to research new development strategies the parallel processing offers. So far majority of the efforts were targeted at the improving the performance and optimization maps of the queries within the traditional physical data warehouse architectures. For example, traditional data warehouse vendors like Teradata joining the Hadoop movement and applying parallel processing to their physical DW infrastructures. Companies like Yahoo and Amazon are also spearheading map/reduce Hadoop adaption for large data scale analytics.

I had been monitoring advances in the Hadoop front in particular, as I believe it will provide a convergence grounds for our products and a new development direction for Queplix Data Virtualization. Data virtualization and Hadoop are born out of the same premise – provide data storage scalability and ease of information access and sharing and I see how the two technologies complement each other perfectly.

Hadoop’s data warehouse infrastructure (Hive) is what we are researching now to integrate with Queplix Data Virtualization products. Hive is a data warehouse infrastructure built on top of Hadoop that provides tools to enable easy data summarization, adhoc querying and analysis of large datasets data stored in Hadoop files. It provides a mechanism to put structure on this data and it also provides a simple query language called Hive QL. Queplix Data Virtualization will soon utilize the flexibility of its object-oriented data modeling combined with massive power of the Hadoop parallel processing to build virtual data warehouse solutions. Imagine the analytical performance of such virtual data warehouse solution created by using the Virtual Metadata Catalog and Virtual Entities in its base as organizational and hierarchal units (instead of traditional tables and columns and SQL-driven access).  Such “virtual” data warehouse solution would be a perfect fit for large scale operational and analytical processing, data quality and data governance projects with full power of Queplix heuristic and semantic data analysis. Today, data virtualization solutions are deployed by many larger enterprises to gain the visibility into the disperse application data silos without disrupting the original sources and applications; in the near future Data Virtualization and Hadoop-based virtual data warehouse solutions will be deployed in tandem to implement the full spectrum data management enterprise solutions ranging from larger-scale data integration projects (i.e. massive application data store mergers as a result of M&A between large companies) all the way to Virtual Master Data Management pioneered by Queplix. Such solutions will not only provide a better abstraction and continuity of business for the enterprise applications but also will utilize full power of parallel processing and will provide immense scalability to Queplix semantic data analytics and data alignment products.

Here are some of new and exciting ideas about Queplix is working on now: utilizing Hadoop for Virtual CEP (Complex Event Processing) within Queplix Virtual Metadata Catalog; generating “data steward” real time alerts using predictive data lineage analysis actually before the data quality problems start affect your enterprise applications; implementing Hadoop-based virtual data warehouse solutions to provide High Availability for large application stores that require massive analytics and semantic data processing; large scale Virtual Master Data Management initiatives involving enterprise-wide Customer or Product Catalog building; large-scale Business Intelligence projects based on Queplix Virtual Metadata Catalog.

Watch this blog for new developments and advances of Queplix technology to integrate Hadoop and Data Virtualization as we make announcements throughout this year!